An overdue status update on Darklang

We've been working hard at Darklang for the past year, but haven't been very vocal about what we've been up to.

Here’s the “Darklang” that’s been live for years:

Darklang – the live version, which we're now calling Darklang classic – is a developer tool composed of a few interconnected parts: a language and interpreter, the standard library, a few package functions, an editor, the cloud runtime, HTTP handlers, support for User DBs and Crons, etc. These parts are written as one cohesive platform for painlessly building cloud-native backends, without dealing with infrastructure, deployments, or any of the other steps between code and software.

It’s been live since 2019, with iterative improvements since. We’ve had some mild success, but Darklang hasn’t quite yet made “product market fit.”

At the start of 2023, we came to a realization that we were being held back by a few things:

• our custom in-browser editor was bad, and disjointed from our users’ “normal” development flows
• supporting backwards-compatibility was slowing everything down
• we weren’t writing much Darklang code, ourselves
• users voiced a feeling of vendor lock-in, due to our license and our-cloud-only runtime
• broadly speaking, everything was incomplete: the language, editor, error-reporting, type-checking, package management. this made it hard to recommend

At the same time, AI-based code generation had quickly become an essential tool for a majority of programmers, despite still being in its infancy. This changed a lot of fundamental assumptions about what Darklang needs to do, especially since it wasn't possible to integrate with Copilot in its current form.

So, we forked the codebase, deleted half of it, and started experimenting.

From Feb to June 2023, we tried a few things, studied and practiced everything related to AI code gen, and eventually settled on what the next iteration of Darklang would be:

• same basic idea: write code, get software, no BS
• abandon our custom editor, write a language server in Darklang, and work with existing editors
• build much more of Darklang in Darklang
• run Darklang code anywhere – in our cloud, your cloud, a local terminal, or anywhere you need to run a script (like CI)
• maintain Darklang's strengths: deployless, trace-driven development, no infra setup
• position the product to integrate with AI where appropriate, code-gen and otherwise
• open-source as much as we can

We’re not quite ready for users, but we’re circling in on some releases.

In the meantime, this post expands upon the limiting factors we recognized with Darklang-classic, reviews some early-2023 experiments towards a "Darklang-next," discusses the on-track work that’s been done in the past year, and gives you a peek at the next few months of Darklang's development.

A quick note on product ownership

At this point, the team is composed of Paul (the founder), myself (Stachu), and Feriel. At the start of the year, Paul has stepped away to lead Tech for Palestine, a loose coalition of folks in tech who are working towards Palestinian freedom. While focused on TFP, he continues to act as an advisor for Darklang as much as he has the capacity for. Since that point, I have taken over when it comes to the day-to-day operations of Darklang, and steering the ship.

We needed some changes

For the past few years, Darklang-classic's status was basically: if you were willing to endure the bugs and limitations, you could build cloud apps really quickly, without having to deal with a lot of the BS typically between code and software.

Here's a demo that highlights many of its strengths:

If you're unfamiliar with Darklang-classic, it might be worth watching that, trying it out, or reviewing the docs. Otherwise, you might be missing some context in the rest of the post.

Our custom in-browser editor was bad

In Darklang-classic, the only way to access or edit your code was to log into a custom browser-based editor. All edits were continuously pushed to our backend as you typed. Those changes would immediately affect your production HTTP handlers, Crons, and Queue-processors. Feature Flags offered you the power to manage these updates, ensuring they meet your standards before releasing to all users.

Our editor lived in its own world, outside of our users’ normal development workflow. Sometimes, this was very convenient - if you wanted to make a quick edit to your code, you could log into Darklang, make your changes, and be on your way without having to set up any local development environment, or wait for a big IDE to load up. In most cases, though, it was just another tool to switch to-and-from as you did development work.

The old client/editor was also often broken. It featured a structual / projectional editor that we custom-wrote for Darklang. I'll touch more on this later, but the point here is that our implementation was pretty rough around the edges, often leading users frustrated that they couldn't just type their code the way they wanted. Too fancy for our own good.

Developing the old editor was also a major distraction from more important work. The point of Darklang isn’t to reinvent a development environment - it’s to remove the accidental complexity that has grown in software development. And here we were, spending time trying to build a fancy new editor, distracting ourselves from that, frequently frustrated when it missed features of our typical tools.

Beyond not being aligned with Darklang’s purpose, it was also pretty costly to develop - the code was poorly organized, and changes needed to be carefully coordinated between our backend and client/editor.

Finally, the editor was ~50% of our codebase, it wasn't written in Darklang, and largely deviated from our product's cohesive nature. Everything else, we could imagine incrementally porting functionality to Darklang, but the editor would forever live in its own ReScript world.

Backwards compatibility was very costly

As Darklang-classic hosts all of our users' code and running apps, we advertised a seamless migration of that code as our language, editor, and infrastructure improved. That came with a very tangible cost – even minor changes required careful coordination between the different components.

In reflection, it was a bit premature to advertise backwards-compatibility, and slowed us down a lot. The OCaml -> F# rewrite in particular would have gone by much quicker if we were more tolerant to breakages.

We weren't writing much Darklang code

Besides the client/editor, which was written in ReScript, Darklang’s source was written almost wholly in F#, and we ended up spending all of our time working with those languages/tools rather than our own.

We did have a few Darklang-classic canvases around to manage our users and their data, and had some DevOps processes built in Darklang, but weren’t writing Darklang daily. We probably would have used it more, but the package manager and general tooling just weren't quite ready even for our own needs.

Broadly, only our users' code was written in Darklang, which meant we weren't sufficiently exposed to the pleasures and pain points of writing software with Darklang.

We saw a few additional opportunities to “build Darklang in Darklang” and even announced an intention to pursue such, but progress there was slow-moving, mostly because of Darklang's overall immaturity.

Users don’t like vendor lock-in.

All Darklang code in -classic ran in our cloud, with no easy escape hatch to port or run your code outside of it. This was paired with Darklang's source-available-but-not-open-source license, fueling a general feeling of vendor lock-in.

Generative AI

While these frustrations were growing for a while, at the start of 2023, ChatGPT happened. And then Copilot. Suddenly a wave of AI-related ChatGPT-wrapper startups took over the news both in the tech world and outside of it. What would development look like in 5 years? We're still uncertain, but felt we likely weren’t positioned for it — certainly, not positioned to empower our users to build in some AI-first way.

So, we forked the codebase

In Feb 2023, we took the Darklang repo, forked it, and started deleting things, just leaving the bits that we felt mattered. First the client/editor and Playwright tests - gone. Then, every bit where the editor’s nuances bled into our backend code. And we continued onward, deleting anything that didn’t fit with Darklang’s goals. It was glorious - over half our repo, gone, and we could start to see things clearly again.

"Darklang-next" or just "Darklang" is the thing we've been building this year, out of that gutted repo.

Meanwhile, we rebranded the old version of Darklang as "Darklang Classic." While still available for use, it’s effectively in "maintenance mode" currently. We are committed to fighting any fires that come up, and will eventually provide a migration path for our users to migrate their existing canvases to Darklang-next, but are not pursuing any new features for -classic.

Early experiments

AI code generation had recently taken over the tech news, and felt both exciting and threatening. We questioned whether our product would fit well in an AI-heavy world, competing against Copilot and the like, and released a post announcing a pivot to reposition the product accordingly.

We followed that announcement by spending the next months doing AI experiments, from Feb to June, figuring out how we'd fit in. Before long, we came to the conclusion that our old Darklang-classic setup indeed wouldn't survive in a code-generation world (not to mention whatever else evolved out of LLMs and other AI advancements).

Those early experiments included chatgpt plugins, a lot of prompt engineering, learning about vector databases, making some conversation-centric chat bots, and imagining of how AI would fit into our overall workflow. Throughout, it became clear that abandoning the old editor was the right move. We experimented pretty wildly, and the fine details are lost on me now, though we had this Notion doc at one point to help organize the ideas we were brainstorming.

The time spent on AI experiments gave us useful context in shaping the product in ways that we feel will pair best with an AI counterpart. Before seriously integrating AI into Darklang, though, we needed to refactor and fill out the rest of the product.

Separate from the AI-heavy experiments, we started work on a CLI (command line interface) runtime for Darklang, as a typed, no-BS seamless alternative to writing bash or python scripts. This was essential to serve as a new foundational layer for our development (as the old editor was gone), while also serving as a strategic step to mitigate concerns regarding vendor lock-in. Mostly, we believe that writing CLI scripts and applications with Darklang will be a much nicer experience than writing them with anything else, fitting the overall theme of the product.

What we settled on – our current vision

After a few months of experimenting with what Darklang could be, having let go of some constraints, a few solid themes/directions emerged.

- same basic idea: write code, get software, no BS
- abandon our custom editor, write a language server in Darklang, and work with existing editors
- build much more of Darklang in Darklang
- run Darklang code anywhere – in our cloud, your cloud, a local CLI, or anywhere you need to run a script (like CI)
- maintain Darklang's strengths: deployless, trace-driven development, no infra setup
- position the product to integrate with AI where appropriate – code-gen and otherwise
- open-source as much as we can

(this is just a copy of the bullets in the intro, as a reminder)

On-track improvements so far

Here’s what progress we’ve made so far towards “Darklang-next,” aligned with those conceptual changes. I try to break down the changes by category - editor, parser, package manager, etc.

Editor

The very first thing we did back in Feb 2023 was throw away the entirety of the old editor. All 200 .res files and 50k lines of ReScript (and a bit of JS), gone.

We've since replaced that custom editor with a language server, written in Darklang, and a thin VS Code extension that wraps it. The language server follows the Language Server Protocol, so we'll be able to eventually support any of these editors (Vim, Rider, Sublime, etc) with a bit of effort.

So far, support is pretty minimal (and not yet worth installing), but we've done the groundwork to keep iterating and expanding, adjusting the language server while using it.

While the old editor is gone, that doesn't mean its old features are gone forever. Within the VS Code extension, we will continue to support deployless development, without setting up infrastructure, powered by your traces.

Parser

Darklang-classic didn’t quite have a parser. At least not one run against user code.

As noted earlier, our old editor contained a custom structural editor we wrote for Darklang, named Fluid. Basically, the "source of truth" for Darklang code is an AST, and every keystroke in the editor made direct structural edits to your AST. We would have to handle scenarios such as “What should happen if a user hits the | key at the end of let x = true – are they trying to pipe (|>) that true into some following code, or are they starting a Bool ‘or’ (||)?" Our implementation of a structural/projectional editor turned out to be pretty buggy, leaving users frustrated not being able to type the code they wanted.

In any case, while Fluid was buggy, it was also the only way to write Darklang code so far, and it was pretty tied to the editor around it. Rather than try to air-lift Fluid into some other editing environment, and fix it from there, we concluded that it was finally time to write a Darklang parser.

Ok – Darklang-classic did have a parser, but it was only used internally, to write test files to be run in local dev and CI. Basically, we had a bunch of .dark files that contained simple tests to be parsed and run.

// backend/testfiles/bool.dark

Bool.and_v0 false false = false
Bool.and_v0 false true = false
Bool.and_v0 true false = false
Bool.and_v0 true true = true
Bool.and_v0 (8 >= 5) (6 <= 8) = true

This 'internal' parser we used to run tests was a hack around F#'s parser – we'd parse the "Darklang" code as F#, and then used the FSharp.Compiler.Service library to "parse"/map that into Darklang code.

Since the big fork a year ago, we've extended this internal parser broadly, using it to build out Darklang. Although it worked for writing tests internally, and it's been "good enough" for local development since the fork, relying on it for more than that is a non-starter:

Since that internal parser is a wrapper around F#'s, our grammar/syntax has been limited largely by F#'s grammar/syntax – for example, end is a keyword in F#, so it'd be difficult to allow that as a variable name in Darklang, as F# would parse it in a special way.

Beyond the restrictions the F#-wrapper parser imposed on our language's grammar, Darklang's runtime is sometimes run in WebAssembly, and there were technical issues around running the F#-based parser in WASM. Specifically, it relied on reflection, a feature of .NET that is incompatible with ahead-of-time compiled WASM.

So, we needed to write a 'real' Darklang parser, from scratch. We had heard great things about tree-sitter, so we checked it out and have been writing our parser with it since. So far, it’s been a fantastic experience. We feel our use of tree-sitter in particular will enable us to build a pretty tolerant parser, which will be helpful for future AI adventures. Our tree-sitter grammar still needs to be caught up with where the hacky F#-based parser is, before we switch over to it fully, but we'll get there.

Package manager

In -classic, most Darklang code was ‘user code,’ floating in the confines of a user's "canvas" – roughly equivalent to a github 'repo.' Users could create functions, scripts, crons, http handlers, async workers, datastores, manage and serve static assets, etc, but all of this was only available on that canvas, unshareable to anyone else.

Separate from code that existed in “user space,” Darklang-classic’s editor showcased a very thin “package manager” that housed a few functions to call external APIs (Twilio, Slack, Stripe, etc).

Underneath package code and user code was a large host of "Builtin" functions, written in our F# source, for core functionality such as String.toUpperCase.

The intent was to open the package manager up for more general use, allowing users to contribute to a central cloud-managed repository of code, housing all sorts of types and functions — anything that’s prepared for wider use. At the time, it was barely used, barely implemented, and wouldn't be particularly useful until our type system would be more built out. The idea is that, if you write code, it's immediately available to you, the rest of your team, and (optionally) everyone else as well. Do you want to share just one type or function with someone? No problem — unless you’re working within a private ‘repo’, they can reference it immediately after you "save" it.

Since the fork from -classic, we've made a lot of progress on this front. We have an actual package manager, it's written in Darklang, and it's used by both our Cloud and CLI runtimes. In fact, almost everything is in package space – except for the essentials still written in F#, most of the standard library is written in Darklang, stored in and served by the package manager. It's centralized along with the rest of 'package space.'

There's still plenty of work that needs to be done on the package manager, and how it integrates into the experience of editing your Darklang software, but we've made good progress so far.

Additional runtime: the CLI

In Darklang-classic, code only ran in our cloud, on our servers, seemingly bound to our editor. While we anticipate our cloud to be the most used runtime, powering your live cloud-hosted applications, we wanted Darklang to be usable outside of our cloud, while still providing a cohesive, seamless no-BS development experience and runtime.

So, we started work on a terminal / CLI runtime. With it, you can write code anywhere, and run it anywhere else, as long as a single darklang exe is around.

Our CLI runtime requires no big installs, and running Darklang code involves no repo-cloning. Just download the single-file executable, provide credentials if the target code is private, and run a command like ./darklang run [fn name].

The Darklang CLI runtime works as a standalone executable, and contains the base language and interpreter, the parser, and access to our package manager. The package manager ends up containing most functionality, including support for local development, as well as running our language server with just ./darklang lsp.

Optionally, you can install Darklang. Once it's installed, you can run Darklang code from anywhere with just dark run [fn name]. The install is minimal, and self-updates seamlessly.

The CLI runtime can run anywhere – Linux, Mac, Windows – both locally on your laptop, and anywhere you need to 'run a script', like CI, or your NAS. We see it as a much nicer alternative to writing bash and python scripts.

There's plenty of work to still do with this runtime, and we're excited to get it all in place.

Language and Interpreter

Darklang-classic had enough of a language to get by, usually, but fell quite short from the intended language. It didn’t even support custom user-defined types – you couldn’t create a Person type and reference it in your code. How frustrating.

Darklang, the language, is intended to be functional, ML-style, and gradually typed. Something like this:

type Patient =
  { name: String; age: Int }
  
type Diagnosis =
  | Illness of String
  | Healthy

let diagnose (patient: Patient): Diagnosis = 
  ...

let diagnoseStr (p: Patient): String =
  match diagnose p with
  | Illness illness -> $"{p.name} is diagnosed with {illness}"
  | Healthy -> $"{p.name} is healthy"

let john = Patient { name = "John Doe"; age = 70 }

diagnoseStr john

Darklang-next has made many many improvements to language and interpreter since the fork.

• We removed null (really we just renamed it to unit and removed everything special about it)
• if expressions no longer strictly require an else component
• Tuples have been added to the language
  let a, b = (1, 2)
• String interpolation is now supported
  $"Name: {name}, Age: {age} years old."
• Darklang-classic had a nasty reuse of one thing for both "Objects" and "Records" – an unstructured Dict type.
  • We now support Dictionaries, a simple data type with keys and consistent value types
    let lookup = dict { a = 1, b = 2 }
  • And we support Record types
• type Person = { age: Int, name: String }
• let joe = Person { age = 18, name = "joe" }
• Relevantly, we now support type-safe record updates
  let olderJoe = { joe with age = 19 }
• You can now create user-defined types. Again, I'm not sure how we got as far as we did without this.
  • abbreviations type RequestHeader = String * String
  • enums / discriminated unions
    • type Option<'v> = | None | Some of 'v
    • let age = Option.Some 1
    • match age with
| Some age -> Int.toString age
| None -> "Unknown"
  • records (shown above)
• We've added support for globally-defined package and user constants
• Darklang-classic only supported a single type of Integer – a 64-bit signed integer named simply Int. We've since renamed that to Int64, and added signed+unsigned Int8, 16, 32, 64, 128. We still need to circle back to add flexible, infinite-precision integers, which will be the default, and name it Int

There are still more language and interpreter advancements to work through, but we're in a much better place than we were a few months ago.

Type-checking

Darklang has always been a statically typed language, with a gradual typing element. At runtime, be tolerant – run as long as nothing in the execution path fails (like JS or Python). At rest, report as many warnings/errors as reasonably possible (like Java or OCaml).

Darklang-classic was pretty far from reaching those goals. Its type system was very basic, not even supporting polymorphism/generics, and type-checking was minimal at best.

Since -classic, we've made some really good headway expanding our type system, as well as our runtime type-checking, which was nearly working, before I abandoned two PRs a few months ago...

Soon, we'll circle back to implementing an at-rest / static checker. We'll be able to do all of the normal code-based type-checking, ensuring you don't try to add a String and an Int. We'll also be able to take unique advantage of Darklang's access to users' trace data, providing you a class of diagnostics unavailable in traditional software languages.

Standard Library

As with the Darklang language, the standard library has evolved quite a bit.

One major sore point of Darklang-classic was our HTTP client – we provided HTTP client functions that tried to be too fancy, doing magic for you so you wouldn't have to think about things like content-encoding and JSON serialization. While the magic worked great sometimes, it often got in your way, confusing users or outright preventing reasonable functionality. Our new HTTP client is much simpler – just let request (method: String) (uri: String) (headers: List<String * String>) (body: List<Byte>) : Result<Response, RequestError> = .... Nice/fancy wrapper functions can be built on top of this, in package space, where you can fork whatever magic as you need.

Another pain point in -classic was JSON serialization and parsing – we provided a simple reflection-based Json.parse, meanwhile JSON serialization only happened automatically, when you returned an object in an HTTP response – this was also reflection-based. Neither the serializer nor the parser provided any way to control how data was stringified/parsed. Darklang-next still offers a pair of automatic / reflection-based Json.parse<'T> and Json.serialize<'T> functions, but also provides a route where you have full control for manually tokenizing/detokenizing your data to/from JSON.

Beyond those cleanups, we generally added a bunch of standard library types, constants, and functions as we needed them – too many to list.

Building Darklang in Darklang

All of the -classic standard library was written in F#, in our source code. In Darklang-next, we've ported the majority of our standard library to Darklang itself – most String and List functions, for example. If something "needs" to be written in our F# Builtins (i.e. for performance) then we will, otherwise we are defaulting to writing stdlib in Darklang.

In addition to porting much of our standard library, many of our language tools are migrating their way to Darklang – the definitions of structured runtime errors, our pretty-printers, half of our parser, etc. Even our LSP-compliant language server is written in Darklang. All of these language tools will be readily accessible to our users (i.e. to fork the language server to override what happens upon onHover, just as easily as they would edit and fork their own code).

The vast majority of our CLI runtime functionality is also written in Darklang - everything from dark help and dark install to dark package view OpenAI.

Tossing the old client/editor in favor a proper, standard language server has provided us with so many more opportunities for 'building Darklang in Darklang' than we previously had available. We're seeing this across each component of our product.

Simpler HTTP Handlers

Like the HTTP Client functions that we previously shipped in Builtins, our logic used to process our users' HTTP endpoints included too much magic. Our code did special things depending on what headers it saw, whether or not a request body was included, etc. The magic, similarly, got in the way and additionally required all requests to have string bodies – no images or other binary data supported. It led to simpler user code but often got in the way.

Our HTTP handling is now much simpler – users' handlers accept a simple Request as an arg, and are expected to return a simple Response – no more magic, and we support binary data now.

type Request =
  { url: String
    headers: List<String * String>
    body: List<Byte> }

type Response =
  { statusCode: Int64
    headers: List<String * String>
    body: List<Byte> }

As with the HTTP Client, middlewares for HTTP handlers will be written in Darklang, available to simplify user code, doing the appropriate magic for your use case, handling auth and CORS and so on. Naturally, these middlewares will have the same seamless debugging experience and forkability as any other Darklang code.

The next few months of Darklang

Technical checkpoint: good enough for CLI dev

Our near-term goals for Darklang are to release what we've been working on, and get an initial wave of motivated Darklang-next users. Specifically, we're just aiming for a version of Darklang-next that's good enough for CLI / local development, connected to our package manager. We'll follow up later, hosting users' cloud apps.

Towards this goal, we need to:

• expand the grammar in our new parser, to match our full language
• finish the package manager, and related work around source-control
• tidy up our CLI runtime
• generally improve the editor until it's "good enough" to work on CLI programs
• misc improvements to: type-checking, performance, etc.
• get it all into users' hands; ask for and respond to feedback

Meanwhile, we'll be working a bit more on our Cloud runtime, but just aiming for that to be "good enough for us" at this point – not quite good enough for users.

Given these advancements, we should be able to "show off" the point of Darklang much more clearly.

Licensing and Sustainability

Today, Darklang's source is available, and users have been using the service freely, but it's not open-source.

Our intended business model at this point is to:

• charge for infrastructure costs, like AWS
• charge for private code, like GitHub

We are going to fully open-source Darklang, including the cloud runtime – our main blocker is ensuring our sustainability, and protecting Darklang the company from being eaten in a few years by the likes of AWS, as we've seen with other OSS projects in the past. We're talking to lawyers and such to figure this out.

Community Focus

Beyond all of these technical advancements, we need to consider all of the non-tech stuff that we've been neglecting for the past year. We are committed to being more vocal, accessible, and make the project more contributable going forward. Specifically, we are aiming to

• coordinate better contrib-accessible discussions of Darklang's progress and planning
• publish more content generally (posts, streams, videos, etc)
• post more regular 'releases' and update posts

Thank you to the folks who have engaged in chatting about Darklang on Discord in the past years, as well as ditansu, nicu-chiciuc, and xtopherbrandt for their contributions most recently.

Additional upcoming goals

Darklang is still a platform for building software without worrying about deploys or infrastructure setup – one cohesive product enabling you to build it all, with trace-driven debugging. With the old editor gone, we have an opportunity to incorporate it all into a package that fits more cleanly in your development workflow.

Here are some steps along the way:

• generally expand our language server to provide an excellent editing experience
• enable editing in additional LSP-compliant editors (Vim, Sublime, ...), including a vscode.dev-like in-browser editor
• bring back our cloud runtime, and avail it to users
• provide a path to transition users of -classic to Darklang-next, and reestablish backwards compatibility
• back to AI: code-generation, AI-centric workflows through voice, and replicate the whole langchain landscape in Darklang
• enable the creation of beautiful modern CLIs in Darklang, inspired by projects such as charm.sh and gui.cs
• bring back other features of -classic: traces, live values, static assets, etc
• continue to improve everywhere: language, type-checking, tracing, debugging, etc.

We'll do our best to keep in touch with this work.

What do you want from Darklang? We'd love your feedback. Let us know:

• in the comments below
• on Twitter, Mastodon, Reddit, Lobsters, or HackerNews
• in our Discord

If you're interested in following progress, follow the email newsletter - we rarely email more than once a month. If you'd like, join our weekly call: we meet every Monday at 11am EST in a Discord voice call, for ~30mins!

P.S. I've written a post on my blog, alongside this one, going over my personal perspective on Darklang: "I'm really excited about Darklang."