The System I Regret Building

Editor’s note:We asked Maya to write the story she didn't want to write. This is it.

In the summer of 2021, I built a job scheduling system that I was genuinely proud of. The team was running a data pipeline platform at a mid-size streaming company, and our existing infrastructure, a collection of cron jobs duct-taped together across three different environments, was failing in ways that were increasingly hard to debug. I spent three months designing a replacement: a DAG-based scheduler with configurable retry policies, resource bucketing by priority class, a clean internal API, and observability built in from the start. It was, by any reasonable engineering standard, the right tool for the problem we had.

The adoption was faster than I expected. Six months after we launched it, seven teams were using the scheduler. A year later, fourteen teams. It had become the default infrastructure for anything that ran on a schedule at the company. I watched this with the particular satisfaction of seeing a system actually used. That was the feeling I should have examined more carefully.

The problem didn't announce itself. It accumulated. By mid-2022, I started noticing that new engineers were reaching for the scheduler not because their jobs required DAG-based dependency resolution, but because the scheduler was there and the pattern was established. Simple tasks that could have been ten-line cron entries became 200-line YAML definitions with retry policies and resource allocations that nobody had thought carefully about. The scheduler's power had become its own gravity.

The architecture had no off-ramp. I had built a system optimized for the hard cases, and there was no friction-free path for the easy cases. Engineers who needed to run a script every morning at 6 a.m. found themselves learning the full scheduler abstraction because that was how you ran things here. I had designed the ceiling without designing the floor.

In Q3 2023, three engineers left the data infrastructure team. In their exit interviews, two of them cited "unnecessary complexity" and "too much overhead for simple changes." One engineer was specific: she said she had spent more time maintaining scheduler configuration than writing the logic the jobs were supposed to execute. She was not wrong. The system I had built was powerful. It was also expensive to operate, in a currency I had not accounted for: the cognitive load of the engineers who had to live with it.

I have thought about what I would do differently. Start smaller, a minimal version that handles the simple case first, even if it means rebuilding when you hit the ceiling. Build an explicit escape hatch into the design: a documented path for engineers who don't need the full abstraction. Make the easy thing easy, and make the hard thing possible, but don't make the hard thing the only thing. These sound obvious in retrospect. Most things do.

What I cannot tell you is whether I should have built it. The teams that genuinely needed a sophisticated scheduler still use it. The data pipelines it manages have run reliably for three years. The engineers who left might have left anyway. I can construct a version of this story where the system was the right call and the attrition was incidental. I can also construct a version where I handed fourteen teams a complexity tax in exchange for a solution three of them actually needed. I don't know which version is true. That uncertainty is the part I didn't expect to be carrying.

The system did exactly what I designed it to do. That turned out to be the problem.