Dark is a holistic programming language, structured editor, and infrastructure, for building backend web services. It’s aimed at frontend, backend, and mobile engineers.

Our goal is to make coding 100x easier, which we believe will allow a billion people to build software. This is a huge challenge, that requires a significant rethink of how we write software.

This post is to talk about how we think about the world. We’re introducing our core philosophy, high level direction, and our goals. We’re going to go into more specifics in future posts (see bottom; or get notified by following Dark or via the mailing list).

Accidental complexity

The defining principle behind Dark is that we want to remove all accidental complexity from coding. Building applications today is incredibly complex, and great deal of that complexity is accidental.

The phrase “accidental complexity” (or incidental complexity) comes from an essay called “No Silver Bullet”. In it, Fred Brooks divided complexity into “essential complexity”, which is core of the business problem you’re solving, and “accidental complexity”, which is everything else you have to do to make software.

He pointed out that in order to improve productivity 10x, we have to remove 90% of the things that we do. At the time, he felt that software development wasn’t 90% accidental complexity, and so no 10x improvement was possible. Hence the name “No Silver Bullet”.

When I look around software today, 90% seems like a low estimate. The amount of accidental complexity is out of control. Look at “It’s the Future”, and “How it feels to learn JavaScript in 2016”. Why do we need to know how to use all these tools to write software?

Fundamentally, writing software is just receiving data, manipulating it, storing it, and sending it somewhere — easy concepts that we learn in our first programming tutorials. Why is it that to build an application today we need to learn Kubernetes, Docker, Git, load balancers, dozens of AWS services, SQL, NoSQL, Kafka, Unix, GraphQL, gRPC, npm, Heroku, DNS, memcached, JWT, Nginx, and the rest of the endless list of tools and technologies that each provide one part of an application?

Complexity is out of control, and it hasn’t shown any signs of slowing down. It’s already at the point where frontend developers feel they can’t write backends. They have so much of their own complexity to deal with that they can’t find space for the backend complexity. How did we let this happen?

A holistic language

The core problem here is that the tools we have been building — as an industry — are incremental. This is natural: you build a new tool to scratch an itch, to solve known problems, and you do it in a way that can be adopted next to users’ existing tools. “Do one thing and do it well” is the core of the Unix philosophy that we’ve been following since the 70s.

Here’s the evolution of common tools in our industry:

• RCS → CVS → Subversion → Git
• Servers → Shared unix boxes → Virtual-private servers → Heroku → Lambda or Kubernetes
• Nightly build script→ BuildBot → Jenkins → CircleCI
• Ed → Vi → Vim → Atom/Sublime/VSCode/etc

Each new tool improves on the previous tool, but fills the same space without removing concepts and complexity. In fact, they often increase complexity: Git is a better tool than Subversion, but Subversion was a lot simpler.

Decreasing complexity

Instead of building a better version of existing stuff, the way to decrease complexity is to entirely remove things.

If we build a tool that does two things, it removes not just the interface, but also reduces the surface area of both problems. If we bring four or eight tools together, we can eliminate huge swathes of complexity, interfacing and overlap between the tools.

Dark is holistic: it combines an editor with a language and infrastructure. It is not a general-purpose language, or a general-purpose editor, or general-purpose infrastructure. Each piece is designed specifically to work with the other pieces, and this tight integration allows us cut out a great deal of that complexity:

• Infrastructure complexity
• Deployment complexity
• API complexity
• Code-as-text complexity

Infrastructure complexity

Infrastructure complexity is everything involved in working with the machines on which we run our code, as opposed to just working with data and customers. This includes Docker and Kubernetes, EC2 and the entire AWS/GCP/Azure ecosystem, dealing with queues, networking, firewalls, load balancers, service discovery, scaling, monitoring, security, DBs, sharding, and optimization.

In Dark we run the infrastructure for you. Because we deeply understand your application — code, traffic, and data — we can make good decisions about the architecture you need. As such, we can cut out all of the infrastructure choices: you don’t need to handle machines, orchestration, storage, databases, or queues.

In addition, we should be able to make great scaling decisions automatically — we think of Dark as an infrastructure compiler, compiling the ideal distributed system for your application. We expect to be able to abstract and simplify in ways that Lambda, Kubernetes or Heroku can’t.

Take databases for example. Today, an application and its database need to be carefully managed together, despite being completely different in terms of operational concerns, languages, formats, and operator skillsets. Combining a database with its application removes many questions that neither component can answer by itself, such as what indexes are needed, how to shard, the best way to write queries, and how to manage long-running data migrations.

Similarly, interfacing with a database from an application is complex: I need to think about whether the types line up, how my ORM might fail, how it will scale, what indexes it needs. Why can’t I just store data, query data, and access data when I need it?

The tight integration of language, editor and infrastructure also allows us to provide observability into your live production systems right from the Dark editor, which completely changes how people code and build systems. It also removes the need for separate monitoring, tracing, instrumentation, logging, and error handling.

Deployment complexity

Deployment complexity is the problem of syncing finished code from one machine to another. This is one of those things that should be trivial, but doing it safely and quickly causes so much trouble that I built CircleCI to help solve it.

Companies dedicate massive resources to reducing the end-to-end time of shipping a change to customers. But the act of shipping code involves necessary steps which take time. Packaging code (Docker container, tarball, webpack, AMI, git checkout, jars), testing it (CircleCI, code coverage, browser testing/Selenium), syncing it (git push to Heroku, Docker registries, artifact hosting, S3, CDNs), enabling the new code (Kubernetes, reverse proxies, Capistrano, swapping symlinks), and rolling it out (feature flags, blue-green deploys, DB migrations, API versioning), involves not just massive complexity and tooling, but also unavoidable wall-clock time.

In Dark, we redesign the deployment process to not require these steps. Instead, deployment is trivial. As soon as you write code in the Dark editor, it’s immediately deployed to production, ready to be enabled with a feature flag. There is no long build process, no tarballs, no containers. It’s a simple diff from your editor to our infrastructure.

Dark is designed to make this process safe. Routes, APIs, and queues are all statically typed, and every function and type is versioned and immutable. Deployment safety features, such as feature flags, unit tests, and powerful database migrations, are built into the tooling, language, and editor. Even though a deploy takes only ~50ms, deployments are significantly less risky in Dark than they are in modern toolchains.

API complexity

API complexity comes from how much harder it is to use an API vs calling a function. Using APIs should be about as easy as calling a function, but we have to deal with authentication, rate limiting, error handling, retries. The way to handle these is different in every API, from auth type to protocol (HTTP+JSON / REST / Protobuf / gRPC / GraphQL / SOAP) to calling conventions, and there are millions of APIs in the world to deal with (and growing).

In Dark, making an API call is as simple as a function call. Our tools for building connectors to 3rd party services handle rate limiting, authentication, error handling, and retries, with great default behaviour. Visibility around costs and failures are built into the editor, as is support for secret keys and Personally Identifying Information.

Code-as-text complexity

Code-as-text complexity is a bit of a catch all. It refers to all the miscellaneous problems we have around how we write code and the tooling around it.

An obvious example is syntax errors. Syntax errors exist because we write code as text, and then ask the compiler to read it, which sometimes it cannot. To never ever have a syntax error again, we must have an editor that doesn’t allow them to be added — one that speaks the language, not just plain text.

This concept can be extended to many parts of the tool chain. We rename functions with our editors’ search-and-replace functions, which don’t understand the difference between a function and a variable; we merge branches with Git, which doesn’t understand the syntax or semantics of our code. We write code in Python or Node or Rust, but also have to consider an entire operating system running under it. We use dozens of text-based tools we barely understand, such as git, bash, grep, sed — because none of our tools actually understand our code.

The Dark language is built hand-in-hand with a structured editor and infrastructure. As such, the autocomplete is aware of your whole program, and syntax errors aren’t possible. Refactoring tools are built in, as are collaboration and access controls. Version control is first-class, and is tightly integrated with the rest of the language, including feature flags, and function and type versioning.

Who are we building Dark for?

Long term, we’re building Dark for everybody — experienced coders, new coders, and non-coders — but we need to start smaller than that. Right now, we’re building Dark for existing developers who want to build backends, targeting two major areas.

The first is backends for client apps, such as mobile apps, or single page web apps written in React, Vue, Angular, etc. If you’re a frontend/mobile developer who is dissatisfied with how hard it is to make a backend, you’ll probably like Dark.

The second is building new services in existing microservice-based architectures. Building a new service means signing up for a future of yaml files, deployment pipelines, 2am pages, zero-day security alerts, and eternal maintenance. If you’re looking at new languages and tools (such as Lambda) to reduce this burden, then Dark should help. That’s what Dark is built for, allowing you to build backend services with little-to-no scaling and maintenance overhead.

Naturally, developers making brand new web startups or side projects should also have a great time with Dark. On the other hand, if you’re building stuff like embedded systems, theorem provers, or machine learning models, Dark won’t support this sort of application for many years.

Dark has a mission to bring coding to a billion people. Our belief is that we need to start with existing developers and expand out: if we make coding for existing developers 100x easier, then we significantly lower the barrier to entry for everyone who’s trying to learn to code, or people in “coding-adjacent” jobs, like designers, PMs, analysts, and information workers in general.

Learn more about Dark

This has been an introduction to how we think about the world with Dark. We plan to release many more details and plans over the next few months as we get closer to a public release. Here’s are some posts we’re working on:

• “The problem with text”
• “How did the world get so complex?” (Rails monoliths vs microservices)
• “The downsides of Dark — open source, self hosting, and potential lock-in”
• “Things like and unlike Dark” (comparison to similar concepts like Eve, Glitch, Smalltalk, or Bret Victor’s work)
• How Dark deploys in 50ms
• “Don’t invent anything new” and “Innovate at the boundaries”
• “Implicit vs Explicit in Dark” (dynamic vs static typing)
• Real problems with functional languages
• “Why Dark isn’t Low Code or No Code”
• “Designing a string type for a Unicode world”
• “A retrospective on our experiments with a graph-based language”
• “Why make our own language?”

If there’s one you’d like us to prioritize, or any questions about Dark, say hi on Twitter. You should also join our beta, and follow us on Twitter or Medium or via RSS, or follow me on Twitter.

Thanks to Corey Montella, Dan Bentley, Evan Conrad, Geoffrey Litt, James Tamplin, Jeff Dickey, Jennifer Wei, Justin Poirier, Mike Plotz, Paul Stefan Ort, Ramin Keene, Russell Smith, Steve Krouse and Vaibhav Sagar for reviewing drafts of this post.