Darklang Release 2

I'm pleased to announce Darklang Release 2.

Release 2 is the first of a new versioning scheme we announced recently. We're doing a "Release" every month, which is just a set of release notes discussing what we shipped over the last month.

Release 2 contains all the work we've released over the last two years. Previously, we posted about the timeline and completion of all this work. This post instead discusses the interesting changes in Release 2 – you may also like to read the entire changelog.

Note that Release 2 is not the everything-is-fixed-and-darklang-is-amazing future version which we once referred to as "v2". We realize this is confusing, sorry! It'll be easier next month.

Async

The most foundational change is that the Darklang runtime is now asynchronous, meaning that your programs will no longer be stuck behind other users' programs making HTTP or DB calls. This was a major source of slow Dark programs experienced by most users, as well as the cause of some minor outages.

Darklang's async implementation is built into the runtime, so you do not need to use an async keyword or function annotation to take advantage of it. All Darklang programs are now automatically async.

New Worker implementation

The worker queues have been rewritten. Instead of using a homegrown queue on top of our database, they now use a cloud queueing service (in this case, Google PubSub). The main advantage is protecting them from occasional downtime, however they also execute more quickly, and a major source of weird, hard-to-diagnose bugs has also been removed.

We have also removed a situation where some cron jobs were run twice if they occurred while the Darklang backend was deploying.

Language changes

• You can now put a lambda in a variable and pipe into it.
• Most error messages are improved, especially for bad input.

Performance improvements

• Several bottlenecks to outbound HTTP requests have been identified and removed
• String::length is now O(1) instead of O(N)
• JSON output is now significantly faster
• When reading strings from a user, the string is only passed over once.
• Traces are now stored in the background, after a request has completed, greatly reducing response time on HTTP requests to Dark programs.
• Dark has moved to more powerful cloud machines, going from Google' Cloud's deprecated N1 machines to extremely powerful T2D machines (2.3x faster)

Operational changes

• There is now a Status page.
• Dark's internal firewalls have been significantly improved
• We have upgraded a lot of internal infrastructure, such as Kubernetes and cert-manager.
• We can now sample traces for large canvases instead of saving every trace.
• Removed nginx from builtwithdark.com requests.
• Moved to autoscaling for some parts of our infrastructure.

Breaking changes

There are a number of breaking changes. I know, Dark is designed to have your code continue to run in the exact same way forever. However, I think these small changes – required to switch over completely to the rewritten backend – were worth it.

Note that all breaking changes were announced in advance, often many months in advance, via the changelog. We  also very carefully deployed the new versions, carefully watching all user canvases and routes to ensure they continued to work the same. As far as we can tell, only 3-4 users had minor issues, which we caught and fixed.

There is only one breaking change that we are  annoyed about, which is that we no longer correctly convert some Unicode characters when calling String::toUppercase and String::toLowercase. This is due to .NET (in which Darklang is now implemented) not supporting "full" Unicode mapping.

Docs improvements

We reorganized the Darklang documentation extensively, using the Divio documentation system. This made it a lot clearer what docs were needed, and where they should go. More importantly, it made it clearer why some docs weren't working, allowing us to separate reference material from introductory material, for example.

We also added new reference documentation about the Error Rail, Workers, and expanded the keyboard shortcuts reference.

Contribution changes

Finally, we made it easier to contribute, by adding coding guidelines, adding significant documentation to the dark repo, and moved 99% of project collaboration to our public github and public community slack.

Read the rest of the changes in the changelog entry for Release 2, or sign up to try Darklang