We Built a System We Don't Understand. Now What?

For most of my career, the way you built a safety-critical autonomous system was settled. You decomposed it.

Perception detects the world and hands over a list of objects. Prediction estimates where they are going. Planning decides what to do about it. Control executes. Each module had a specification, and because it had a specification, it could be tested on its own, verified on its own, and trusted for a stated reason. The whole discipline of functional safety rests on exactly this: specify, decompose, allocate, verify each part, compose the argument.

That architecture is losing. In self-driving, the hardest problem I have worked near, the expert consensus has been drifting for years toward one conclusion: the systems that work best do the whole chain, sense, plan, act, in one learned model. Not because anyone loves monoliths. Because the module boundaries were where the information died. The perception module had to compress a rich, ambiguous scene into a tidy list of objects before the planner ever saw it, and everything that did not fit the list got thrown away. The pedestrian's hesitation. The body language of a cyclist about to swerve. The thing you cannot name but a good driver reacts to. An integrated model keeps all of it, and it turns out that keeping it matters more than the cleanliness of the architecture.

So end-to-end wins on capability. I accept that. The question is what it does to everything else, because the price is larger than most of the people celebrating it seem to notice.

Every discipline I was trained in assumes the thing we just removed. How do you build redundancy into a system whose function you cannot state? How do you size and qualify hardware for software whose behavior you do not fully know? How do you write a safety case for a function nobody can specify? Redundancy, hardware architecture, verification: all of them start from a specification, and the learned model removed the specification at the root. We did not make the old methods harder to apply. We removed their input.

And this is not a self-driving problem. It is the structural consequence of learned end-to-end systems, and it will surface in every domain that builds one. Industrial control, logistics, energy systems, medical workflows. Anywhere someone replaces a pipeline of specified components with one model that ingests everything and acts, the same question follows them in: how do we assure a function we can no longer write down?

Start with redundancy, because that is where the trap is most concrete. The classical answer to an unreliable component is a second one. Two channels, and one catches the other, because they fail independently. That assumption quietly breaks here. Two copies of the same model are not two opinions. They fail identically, on the same input, at the same moment, because they are the same thing. Duplicating the model gives you twice the hardware and zero additional safety. Real redundancy now requires diversity in kind: the learned model, plus a watcher built on fundamentally different principles, one that fails differently precisely because it is different. Which means the watcher cannot be another copy of the intelligence. It has to be something else entirely.

That sounds like a dead end until you notice what the watcher actually has to do. It does not have to drive. It has to stop bad things from happening, and those are very different jobs. "Drives well" cannot be written as a specification. But "do not collide with anything", "never exceed this deceleration", "stay within the lane boundaries", those can.

The function is unspecifiable. The safety envelope around it often is not.

So the engineering discipline does not die. It moves. Away from the content, which we can no longer specify, to the boundary, which we still can. The watcher does not need to understand driving. It needs to enforce a short list of invariants, and enforcing invariants is exactly the kind of problem our old methods were built for.

The hardware follows the same logic. You cannot size a compute platform against a function you cannot describe, but you can bound the envelope: worst-case compute, latency, power, the physical limits of the actuators. And you can do the one thing that actually matters for the safety argument: give the watcher its own path. Its own compute, its own sensor feed, its own power, so that whatever takes down the intelligence cannot take down the thing checking it. We stopped engineering the content. We still engineer the container, and the container is where the guarantees live.

Now a correction to how this usually gets told, because the success of end-to-end is being generalized into something the evidence does not support. Self-driving did not move to one model that runs everything. It moved to one model that owns sense-plan-act for a single bounded task with coherent physics: driving the vehicle. Above it sits a separate, independent safety layer that can override it, and that layer deliberately does not share a brain with what it monitors, because a watcher that shares a brain with the watched is not a watcher. And the model does not also run the logistics, the invoicing, and the HR system. Consolidation won inside one task. It stopped at two lines: the line to the independent safety layer, and the line between different tasks. The dream of one model that ingests everything and controls everything is not the conclusion of the self-driving story. It is an overreach of it, and it erases exactly the two boundaries that made the story survivable.

There is a harder limit underneath all of this, and I want it stated clearly because it is the part that keeps this honest. The whole move, from specifying the function to enforcing the envelope, works for one reason only: the envelope is simpler than the function. "Do not collide" is radically simpler than "drive well". That gap in difficulty is what makes the watcher buildable at all. But the gap is not a law of nature. There are domains where the safety question is exactly as hard as the function itself. Is this medical recommendation harmful? That is not simpler than "is it correct". It is the same question wearing different clothes. In domains like that, there is no short list of invariants, no simple checker to build, and a watcher capable of judging the output would be as opaque as the system it judges. You are back where you started.

So the method has a boundary of its own. Where safety can be stated more simply than capability, we know what to do: build the envelope, build the diverse watcher, engineer the container. Where safety is as hard to state as the capability itself, we currently have no guardrail that is not made of the same unverifiable material as the thing it guards. Those are the domains where end-to-end consolidation should frighten us, and where the honest answer to "now what?" is: we do not know yet. That answer should be allowed to stand. Pretending otherwise is how this turns into sales material.

One more objection deserves a response. The research frontier is genuinely moving on understanding these models. Interpretability, formal verification of networks, runtime monitoring. All real, all improving. But the old discipline required something close to complete understanding before you bet lives on a system, and partial understanding at research maturity is not that. So the design stance has to be: assume the box stays black. Every piece of understanding that arrives is a bonus on top of a safety argument that did not depend on it. If your safety case needs interpretability to work, you do not have a safety case. You have a hope with citations.

The car will get better, and the model inside it will remain something nobody can fully explain. Both of those can be true at once, and living with that combination is the actual engineering task of the next decade. Capability comes from the inside now. Assurance has to come from the outside. Keeping those two separate, and refusing every architecture that blurs them, is what the discipline looks like after the specification is gone.