Building Dark in Dark

For years, we've talked about building Dark in Dark. In fact, one of the most common questions we've had about Dark is why isn't it built in Dark?

When you are talking about building a language in itself, typically it means a compiler being written in the language, then compiling itself. This is sometimes called "self hosting" or "bootstrapping". But Dark doesn't really fit that model, partially because Dark is a lot of different things (language, editor, cloud, package manager, API, etc), but also because Dark is at the moment interpreted, so there's nothing to compile.

This is really unfortunate, because we would like to spend a majority of our time writing Dark code. Not only is writing in Dark fun, but Dark also solves a lot of the problems of the rest of our stack (e.g. we have a deployment pipeline with Kubernetes and we build Docker containers and it's all kind of slow and annoying).

API

However, we do have an API that's written in F#. The API really forms the backend to the editor, and does everything from updating Dark programs when you type in the editor, to managing traces and secrets and static assets deployments.

A couple of things have prevented us from rewriting this in Dark. The first thing is that we lacked some features. For example, until the recent rewrite, the Darklang runtime wasn't asynchronous. This made it unsuitable for long running requests, such as uploading large static assets.

The second, and really the main thing, is fear. While writing code in production is kind of what Dark is about, most of our users have things a little bit easier than we do. If they break some of their code, they can just change it back. But if we break the ability to change code, we might be unable to recover it (or at least recover it quickly or easily). This could result in a large (or potentially permanent) outage!

I think this is a valid fear. However, it was preventing us from really writing any production code in Dark at all! We do have the onboarding service, the presence service, some stuff around invites, but that's not much stuff at all, and they rarely change.

Going forward, we've made the decision to write anything like this – that could be written in Dark – in Dark.

New services in Dark

There are two main services that we're in the process of rewriting in Dark.

The first part is the static assets service. Static assets is the feature that allows you to host your frontend on the Darklang CDN. It was originally written in directly in OCaml because Dark didn't support many of the features that we needed to implement it in Dark, including async, uploading binary data, making HTTP-requests using binary data, and libraries for Google Cloud. So now, we have some of those features, and we're in the process of building the others.

The other major feature is reimplementing the APIServer. The APIServer is the backend of the editor, providing all the API routes used when editing a Dark program. There's no actual real need to redo the APIServer, however there is a real need to dogfood and since I primarily work on the Dark implementation, it would be much better if it was actually written in Dark.

Yak shaving

Of course, while we do have the goal of actually building things with these projects, the main goal of all this dogfooding is really yak shaving. In a sense, the goal of building new features in Dark is as much to get experience with a current feature set (and especially to find out what's missing) as it is to actually build the new feature.

And so we're also going to be aggressively yak shaving. That means when we run into a problem using Dark, we're gonna stop what we're doing and fix that. When the new feature were building needs another feature in order to be built right, we're gonna build that new feature first.

Essentially, we're going to use yak shaving and dogfooding in as our primary ways of determining want to build and when.

Principles for yak fooding

Yak shaving is typically something that people try to avoid, because it comes with a bunch of downsides. Given that we're actually using this to drive our work, we made an initial set of principles that we think will help guide us in the right direction.

Keep the overall number of yak shaves low

Having multiple projects going at once is an easy way to complete none of them. It's also very frustrating, both to not complete projects, and to be balancing so much in your head at once.

So keeping the number of yak shaves low is important. We are actually trying to ship fixes and features here, so closing out projects, and especially removing the mental load of half done work, is a priority.

Prioritize work that can be completed using existing features

A good starting point is work where there aren't any known yak shaves involved. As we go to implement the work, we are likely to find yak shaves anyway, so better to have zero yaks at the start.

Don't compromise features

The goal of the activity is to ensure that we do things right. This means not implementing shortcut features that will be hard to remove later. Our goal is to expand the actual feature set that we want users to use. We should be thinking about how to complete core features without compromise.

Implement only as much of the feature as is needed now

This doesn't mean we need to complete an entire future. Features need iteration, they need user feedback, they need experience of using them, etc, before we can get them to their final state. In addition many features have many facets to them, and not all of those facets are useful now. So implement only as much of the future as needed as is needed, perhaps even keeping the feature behind a future flag for now (for example if it's not terribly useful to users without several more steps, but what we need internally is already finished).

When faced with two yaks, shave the smaller one first

Yak shaves beget more yak shaves, and a breadth-first search is going to lead to a lot more context in our heads at once than a depth-first search.

Stop shaving two yaks deep

Two might not be the right number here, but at some point we're gonna have to cut things off. Really it depends on how exponential the actual yak shave is. If each yak generates two more yaks, then we may need to keep the depth even lower. However if it's less, it might be safe to go a little bit broader. I think we'll learn this from experience.

Don't forget to remove old features

When you implement new features, don't forget to remove the old features that they replace. Otherwise, that just leads to more code to keep in our heads. Indeed, one of the reasons for rewriting the static assets service, is that it's one of the last things that is written in OCaml, and we would of course like to remove OCaml from our codebase.

Contributors

Finally, as we're building, we're trying to make sure that our paths can be followed by other contributors. If we're improving Dark by fixing our pet peeves and annoyances, we want to make sure that you're able to as well.

Most of our work is tracked in the issue tracker. If you haven't checked out Dark recently you can sign up here or sign in here.