System Design Has Changed: What Actually Broke, and What Still Holds

The old system design we knew is dead. That, at least, is the prevailing mood. It deserves a more careful autopsy than it usually gets, because the death notice names the wrong victim.

The toolkit is fine. Load balancers, caches, queues, replicas, circuit breakers: these patterns absorbed decades of production failure and came out the other side as the closest thing our industry has to settled knowledge. Put an LLM-powered product into production today and roughly seventy percent of what keeps it alive is still this inheritance. Nothing about transformer architectures made statelessness a bad idea.

What died was quieter and more consequential: three assumptions that sat so far beneath the toolkit that most of us never articulated them. We didn't design with them so much as inside them. A language model in the request path breaks all three, and nearly every strange incident in AI systems over the past few years, from viral chatbot screenshots to six-figure surprise bills, traces back to one of the breaks.

The first assumption: identical inputs produce identical outputs

Every discipline we built for verifying software rests on repeatability. A function is tested by asserting its output; a bug is confirmed by reproducing it; a regression is caught by rerunning what passed yesterday. Determinism is so foundational that we don't call it an assumption. We call it testing.

Language models do not honor it. Generation is sampling, and sampling involves randomness by design. Lowering the temperature narrows the distribution but does not close it; provider-side model updates, batching effects in inference infrastructure, and small variations in context finish the job. Two identical prompts, minutes apart, can yield materially different answers, and no configuration flag fully prevents this.

The naive response is to treat this as flakiness and retry until it goes away. The mature response is to change what "correct" means. Production AI teams have converged on a discipline that looks less like unit testing and more like statistical quality control: a curated set of representative cases, criteria for what an acceptable answer must contain, and a score tracked across every prompt revision and every model upgrade. Alongside it, an architectural principle: the model's output is a proposal, and deterministic code, schema validation, policy checks, and idempotent downstream operations decide what actually happens. The model suggests; the system disposes.

None of this is exotic. It is the same instinct that gave us input validation, applied to a component that happens to sit inside our own trust boundary.

The second assumption: requests cost roughly the same

Twenty years of web architecture taught us that requests are cheap and interchangeable, and that cost is something the finance team reviews monthly. Both halves of that lesson are now wrong.

An LLM call is metered per token, input and output, and the variance is enormous: a one-line answer and a document-heavy analysis can differ in cost by three orders of magnitude while traversing the same endpoint. Conversation history compounds the problem, since each turn re-transmits, and re-bills, the turns before it. And the failure mode is novel: a system can multiply its own costs while functioning exactly as designed. An agent retrying a failing step in a loop is not down. It is expensively, diligently up.

The consequence is that cost has migrated from the invoice into the architecture diagram. The patterns that address it are recognizably descendants of the old toolkit. Tiered model routing sends easy requests to small models and reserves capable ones for the work that needs them, which is load balancing by difficulty rather than by server health. Semantic caching answers repeated questions without repeated inference, which is the cache pattern extended to match meaning instead of bytes. Context management treats the model's input window as a constrained budget rather than an attic. And hard per-session spending caps do for money what timeouts always did for time.

A design review for an AI feature that does not include the question "what is the worst-case cost of one session, and which component enforces it?" is not yet a design review.

The third assumption: data and instructions are separable

This is the deepest break, and the one with the sharpest edges.

Software security has one load-bearing wall: code is code, data is data, and the platform keeps them apart. SQL injection was a generation-defining vulnerability precisely because it breached that wall, and the fix, parameterized queries, worked by making the separation structural and non-negotiable.

A language model has no such wall. The system prompt an engineer wrote and the message a stranger typed arrive as the same substance, and the model weighs both. In 2023, a car dealership's chatbot was persuaded, in plain English, to agree to sell a Chevrolet Tahoe for one dollar. In 2024, a Canadian tribunal held Air Canada liable for a bereavement refund policy that existed only in its chatbot's imagination. Neither incident involved a compromised system in the classical sense. The systems did what they were built to do: read text and respond helpfully. The text was the attack.

It grows subtler. Instructions need not arrive through the chat box; they can wait inside anything the system will eventually read, a document in a retrieval index, a page a browsing agent visits, the output of another tool. OWASP now maintains a Top 10 for LLM applications, and prompt injection occupies the first position, but read further down the list and something reassuring emerges. Improper output handling is injection with a new entry point. Unbounded consumption is rate limiting with a wallet attached. Excessive agency is least privilege, unlearned. The threat landscape is genuinely new; the instincts required to face it are not. Defense here looks like defense everywhere: layered controls, minimal permissions, untrusting validation of everything the model produces, and human confirmation ahead of anything irreversible.

For those of us who work in security, there is a certain dark comfort in this. The industry has been arguing for decades that trust boundaries deserve more respect. Now every architecture with an LLM in it has a trust boundary running straight through its center, and the argument makes itself.

What this means for the people learning it

I co-run engineering communities, and the question I hear most often, usually from engineers a few years in, is whether deep system design knowledge is still worth acquiring, or whether the field has become a moving target not worth the chase. The question deserves a precise answer, because the imprecise ones are doing damage in both directions.

The precise answer: the fundamentals have never been worth more, because they carry the majority of every AI system in production, and the genuinely new material is smaller than it looks. It is three broken assumptions and the design patterns that answer them. That is a bounded syllabus. It can be learned deliberately, in about a week, by someone who has never shipped an AI feature.

I wrote that syllabus down. It is a free seven-day course on GitHub, starting from day zero, with no prerequisites; each concept is taught through a production incident rather than an abstraction, and it ends with a project brief that inverts the usual demo: build a support assistant whose interesting property is not that it answers questions but that it cannot exceed its budget, cannot be talked out of its policies by a document planted in its own knowledge base, and cannot loop forever. The chatbot takes an afternoon. The cage around it is the education.

The course lives at github.com/NewAi25/system-design-ai-era-7-days.

One more thing, for anyone hesitating at the edge of this field: it is three years old. Nobody has a decade of experience in it, whatever their title suggests. The assumptions broke for the staff engineer and the newcomer in the same quarter, and the distance between them has rarely been shorter.

The old system design we knew is not dead. It has been given a successor to train. The engineers who understand both, the inherited toolkit and the three new fault lines, are the ones who will design what comes next.