Recent Posts
-
While Postgres Redlined, Robinhood Sharded to Scale
Read more…The alarm went off at 6:20am. I rolled out of bed, opened my laptop, and pulled up Grafana. Equity markets open at 6:30 and I hoped last night’s fixes would hold. I watched request and order counts, comparing them to a week ago when things ran fine. Slack was already buzzing with engineers also checking metrics.
The clock flipped to 6:30. The first metric tick came in, spiking the graphs. Okay, not bad. Next ticks pushed them higher, but error rates didn’t budge. It just needed to stay steady for a minute, then ease off. This could’ve been any morning from 2014, when we started bringing in external Robinhood customers, to 2024, when I left. I was an engineer and later a manager, mostly on brokerage systems.
-
Sroto: Protocol Buffers Evolved
Read more…In my previous post I argued that the existing tools available to us as developers largely either optimize for developer velocity or scale, particularly when build backend applications. I postulate that if we had a framework that derived as much as possible from a schema definition plus associated metadata, that framework could optimize for both. To enable this, we first need to have a better way to write schemas.
Introducing sroto, a library and associated command tool to generate
.protofiles (and thus protobuf schema definitions) from jsonnet, a data templating language.This basic problem with the way we approach schemas is:
- A schema definition is just data.
- Writing data is annoying and we want to write code.
- So we use the application code to write the data.
- Whoops, now we can’t untangle it and therefore scalability is hard.
So let’s break this cycle.
-
How We Can Beat the Tradeoff Between Developer Velocity and Scalability
Read more…I’ve been working at Robinhood for a number of years since the early days and one phenomenon I’ve been fascinated by is the dramatic decrease in developer velocity as both the engineering organization and production systems have scaled.
Here’s an example inspired by real-world events building out the backend API for showing dividends in-app in 2014:
-
DigitalOcean App Platform
Read more…I’ve been quite impressed with DigitalOcean’s App Platform service. The team there struck a good balance of providing some extensibility while not going so far as to undermine the core value proposition of the service: simplicity. As a quick example: this blog is written using Hugo. It’s hosted on GitHub, and using the DigitalOcean admin console to start hosting the blog was a breeze (and it’s free!).
This said, the more interesting use cases of App Platform comes when it’s combined with Pulumi (I’ll assume some familiarity with Infrastructure-as-Code solutions and how Pulumi differentiates itself). To get started, here’s the Pulumi Python code that’s used to host this blog:
-
Map Unicode to ASCII
Read more…Interfacing with older APIs that don’t support unicode can be a pain. Too often I’ve seen errors from third parties saying:
Error: characters must be in the range 0 to 127.Suppose that we’re writing an API for some site that internally will be interacting with these types of third parties. This API will allow people to create users and sign up. We want to allow people to sign up with their real names with accents and all (such as José), but still only give third parties ASCII characters.