While Postgres Redlined, Robinhood Sharded to Scale
The alarm went off at 6:20am. I rolled out of bed, opened my laptop, and pulled up Grafana. Equity markets open at 6:30 and I hoped last night’s fixes would hold. I watched request and order counts, comparing them to a week ago when things ran fine. Slack was already buzzing with engineers also checking metrics.
The clock flipped to 6:30. The first metric tick came in, spiking the graphs. Okay, not bad. Next ticks pushed them higher, but error rates didn’t budge. It just needed to stay steady for a minute, then ease off. This could’ve been any morning from 2014, when we started bringing in external Robinhood customers, to 2024, when I left. I was an engineer and later a manager, mostly on brokerage systems.