Dark's new backend will be in F#

Paul Biggar -

Welcome HN! Dark is a programming language, structured editor, and infrastructure—all in one—whose goal is to make it 100x easier to build backend services. Check out the website, our What is Dark post, and How Dark deploys in 50ms for more. Thanks for checking us out!

Part of a set with Leaving OCaml and Why Dark didn't choose Rust.

Nothing in my life so far would have prepared me for the fact that I would be willfully choosing to move to .NET, but it's 2020, and nothing matters anymore.

I've been evaluating new languages for the Dark backend over the last 2 months or so. For a bunch of reasons, OCaml has been a little unsatisfactory.

Over the last few years we've always said "when we rewrite the backend", "at some point we'll rewrite and this will go away", etc. There's a lot of new code to be written on the backend, to meet our roadmap. Are we really going to write it once, and then rewrite it later? Or would it be faster to just port it now, and write the new code in the new stack?

Ultimately, I decided that if there was going to be a change, now was the time. And more importantly, if there wasn't going to be a change, now was an excellent time to fully commit to OCaml, and not be second guessing the choice.

Initially, I expected to go to Rust. Rust has excellent tooling, great libraries, a delightful community, etc. But after spending about a month on it, I can't say I like writing Rust. Especially, I don't like writing async code in Rust. I like a nice high level language, and that's kinda what you need when you have a project as big as Dark to build. And Rust is not that. I'll publish "Why Dark didn't choose Rust" next. Or I might call it "you'll never believe just how much a Garbage Collector does for you!", because that's the summary.

When I was working on Async benchmarks before, what I was really doing was evaluating "are any of these OCaml alternatives better? And if so are they also faster?". I evaluated and expected not to like F#. I actually quite like it. It's close enough to OCaml, has great library support, and tooling which so far has been a mix of great and terrible. The 90's Microsoft tooling is still there, and that bit isn't all that great, but overall it's a much better situation than OCaml or Rust.

Why did I chose F#?

Let's start with the obvious, F# is OCaml. It's OCaml backed by the world's largest and most experienced creators of programming languages. And in the areas that OCaml is great, F# is also great! Sum types, static typing, eager execution, pipelines, immutable values, all of this is really great.

It actually has a much better type system, in my opinion. One thing that sticks out is that OCaml made it really cumbersome to use maps. Like, hashtables, associative arrays, whatever you call them. It seems the old version of Real World OCaml has been taken down, so I can't show you how unpleasant they were at the start. Now, in the latest version, they are more moderately unpleasant to use. Whereas in F#, you have a Map<OneType,AnotherType> and that's really it. Magic!

Of course, the main reason I chose .NET was the libraries. It has libraries for everything, what a surprise. While there aren't all that many F# first-party libraries, every vendor out there has a .NET SDK that you can use directly from F#. I look forward to finally having first-party support for Honeycomb, Rollbar, and Google Cloud. I'm even finally going to get to use LaunchDarkly, after years of telling Edith I would.

The other thing I've really enjoyed is how good the docs and community content are. A lot of OCaml community content is on the language and what you can do with it. F# developers seem to just want to get shit done. There's a million blog posts, youtube videos, etc, from enterprise software developers talking about the best way to build web software. And then of course the massively detailed and useful FSharpForFunAndProfit, as well as Tomas Petricek's work - it's really great.

I think most of this is due to the size of the community. I've heard people say that there are very few F# developers; something like there's 1M C# users, 100k VB users, and 10K F# users, or something like that. I'm not sure exactly what counts as user, but I would imagine that OCaml has fewer than 100 "users" in this metric, so it feels like I'm moving to a massive community.

Not everything is amazing though. The build system is attrocious. While paket is roughly on par with esy, msbuild is 1000 times worse than dune. An incremental build in dune is like 1s for me, and 6s in .NET, even if nothing is happening. I know they have fancy incremental compilers for .NET so this puzzles me; if anyone has tips on getting really fast compilation in F#, I would appreciate them.

An important thing to check was whether I could compile my code to JS. Dark's execution engine runs it the editor as well, and that's one of the core things that makes Dark special. Because of this, I want to take unaltered backend code and compile it directly to JS. This isn't like Rescript (which is OCaml compiled to JS with slightly different semantics and ecosystem, or it's equivalent in F#, Fable). Fortunately, F# code can be compiled to Wasm using Blazor. Blazor compiles the .NET runtime to WASM and runs your code in that. It barely took any code to get working either (though figuring out the right incantation was not trivial).

Feedback from "Leaving OCaml" blog post

Yesterday's post got a lot of people to add their opinions. In addition to some Scala and Erlang love, a lot of people pointed to F#:

So...I think a decent choice to make here is to switch from OCaml to F#. You'll get almost all of the benefits and most of the drawbacks go away. And for the most part, you can directly translate the code from OCaml to F#. - darksaints
I've been using F# on GCP in production for 3 years now and it's fantastic and only getting better. You can leverage existing .NET libraries (for example, you get official GCP libraries from google) and if you use them enough it's easy enough to write a functional wrapper around them. - angio
We have considered OCaml but went for F# instead. I could not be happier. Great libraries, good tooling, in 2020 F# is a first class citizen in the cloud thanks to C# and .NET Core. You can develop on Linux, Windows, MacOS and without modification it works. Compilation times are great. Unless you want to deal with the low level nature and the C++ influence in Rust F# is a much more logical step to move from OCaml. There is dotnet fsi for REPL too. F# has access to C# libraries and it is relatively easy to write a thin wrapper that convert nulls to Nones, so you are null safe or C# style functions to ML style functions so you can use |> or <| etc. - l1x

Next steps

I recently merged the first F# code into the codebase. It's an async version of the core of the Dark interpreter, connected to Giraffe (F#'s low-level interface to .NET's Kestrel web server).

My plan is to reimplement the same language, but with some of the learnings that are in the roadmap. For example:

  • Result and Options will be types, not special cased into the language
  • Int will be infinite precision (this is technically a breaking change, but not a significant one AFAICT)
  • breaking out the API server, using F# builtin middleware libraries
  • refactoring the Dark public HTTP server into composable middleware, available from within Dark

I'll be working on making the equivalent to the existing Dark interpreter, keeping the semantics the same. After that, I'll be making new features from the Dark Roadmap.

Implementation plan

Dark's backend is 37K lines of OCaml, of which 8K lines are tests, and 10K lines are the Dark standard library. So there's about 20K lines to port. Should be fun. My implementation plan is to get to parity - there's plenty of time after that to take advantage of the ecosytem:

  • finish the interpreter
  • connect Dark's F# backend to Dark's DB, so it can read the same data
  • compile Dark's F# interpreter to JS (done, see above)
  • make an OCaml library to deserialize the binary opcodes in the DB and covert them into F# (via, I presume, a JSON intermediary). This will allow incoming requests to (optionally) be routed to the F# service.
  • add a fuzzer to ensure the semantics of the OCaml and F# versions of Dark are the same (especially the library functions)
  • do all the devops (getting it into k8s, etc)
  • add the APIs we have available in OCaml
  • add a way for users to safely try their code in the new version
  • port over 170 or so functions we have in the standard library
  • start recording traces using the new design (this is actually the main thing driving the whole change, believe it or not!)
  • use Honeycomb, Rollbar, and Google Cloud using their .NET native interfaces
  • port our various services to F# (lower priority)
  • port our accounts to self-managed from Auth0
  • translate the contributor docs to F#

There's lots to do, and Dark contributors are welcome to take part. Porting standard library functions might be an easy place to get started, as is adding language features. Most things are a straight port from OCaml, except as mentioned above. As always, feel free to ask in the Slack or issues if you've any questions.

You can sign up for Dark here. For more info on Dark, follow our RSS, follow us (or me) on Twitter, join our Slack Community, watch our GitHub repo, or join our mailing list.

Thank you JaggerJo for the header image!