Dark devlog #1: fresh start
So right now Dark is made up of 100,000 lines of code, a hosted service with 6000 users, a 1600 member Slack community, 39 people who have contributed or intend to contribute to our repo, and 500ish "real" projects.
And—as of recently—just me to build it.
This devlog is a weekly post where I try to organize my thoughts around how to make Dark successful from here on. It's going to be barely edited and I can't promise it'll always be coherent.
My short-term priorities right now are:
- help the team get new jobs
- reduce costs and complete the transition to a smaller team
- improve stability of the infrastructure
- engage contributors
- improve onboarding
If you're hiring, please take a look at the Dark team looking for new roles. They're great engineers/designers/product folk and they would make great additions to your company!
Improving stability is a interesting one. I'm not really an infra engineer, but I guess I'm about to become one. I just bought the Google SRE book and Charity Majors' Database Reliability Engineering.
I'm the only one on the pager, so I need to figure out how to make this thing stop going off. We've had a few recent incidents that I need to go through to shore things up, and there's one active ongoing incident that needs to be handled.
Right now, our queues are going a little bit nuts due to some intense activity that repeats every day, causing our DBs to slowly increase to 100% CPU over the course of about 12 hours, and then when it hits 100% it stops and everything goes back to normal.
So far I don't know what it is, I don't seem to have enough information to figure it out just yet. I've been rooting through every graph we have in Honeycomb to try to figure this out. So far I've disabled a process we used to keep the databases from blowing up in size, and I that might be the cause, so I'll see what happens today.
Isolation / Rust
Once I've got that done, I need to work on isolation and abuse prevention. Isolation is tough because we use OCaml which is single threaded, and because we didn't use the fake-multithreading libraries (Async and LWT) in enough places. So one option is to do that, and another option is to put some Rust as a control plane in front of the OCaml processes to kill them effectively. Or maybe both.
We have a bunch of operational stuff that comes from having our code in OCaml, so the move to Rust has been long sought. Maybe now I have the time to do some of this.
I've been vaguely thinking "oh I have some time, I could finally do the Rust rewrite I've been talking about" but that seems a little bit like madness. At the same time, I've been having lots of "second-system syndrome" thoughts like "oh, if I rewrote it in Rust, I could make the HTTP stack much more usable" and similar. I probably need to tamp that down a bit.
We have a ton of contributors who have written a PR and are interested to do more. We don't have enough issues for them, and don't have enough contributor docs explaining how to fix more complex issues, so I'll be working on those in the near future too. And of course reaching out to folks to see what I can do to make contribution easier.
More importantly, though, is trying to help make a long term community, where the product and how we build it is centered on the community. It's a bit unusual to go from a team of 10 building a product, to this community model, so I don't expect we'll figure this out overnight. Definitely something I'm looking for input on!
It's still quite challenging to get started with Dark, and while we have some docs and some in-product tutorials, there's a ton to do still. I'll be focusing on this once I've got a handle on operational work.
The most important thing that I need to focus on is Product Market Fit. It's pretty clear that we haven't yet reached Product Market Fit with Dark, and that we're probably not all that close. This is a place where I'm spending a lot of thought.
It's clear that extremely Dark is good at a few things: it's probably the easiest way to put an API on the internet. It's extremely easy to have datastores and queues and HTTP handling.
The most obvious thing is to find something that we're close to, and try to get PMF with that. Should we try to pare down to be "Zapier but with a programming language"? Or really focus on the niche of React SPAs?
Part of the problem with Dark so far has been that programming is pretty general, and so it's been hard to make Dark really great for a tiny niche. It's very hard to predict what integrations and features people need for any app. After all, what do people build with React? Well, just about everything.
Similarly, it's very hard to support simple verticals. You can't be "the programming tool for the travel industry", it doesn't make sense.
A related problem is about how we position Dark. "Dark is a cool new way to program" doesn't tell anyone about Dark or how to use it. We tried to position very small at one point--just Slackbots--but there's not that many people trying to build Slackbots on any given day.
No solutions here yet, but this is what I'm thinking.
I'm new to dev.to, but I'm planning to write here every Monday, mostly stream of consciousness about what's going on with Dark. Would love thoughts and comments here or via twitter or in our community slack.