You Can No Longer Prove It Works

When I started working in computer vision for driver assistance systems, evaluation was the easy part.

Not easy as in trivial. Easy as in well-defined. The system had one job: detect the pedestrian. We had recorded data with the correct answer marked in every frame. Run the system, compare its output against the truth, count. Detection rate, false positives, green or red. If someone asked whether it worked, we had a number.

I did not appreciate at the time what a luxury that was.

The question "does it work?" only has a clean answer when the requirement and the behavior live close to each other. Detect the pedestrian. Classify the sign. Measure the distance. One task, one output, one truth to compare against. You could write the requirement down and hold the system to it.

Then the systems got better, and the requirement and the behavior started drifting apart.

A modern system does not produce an output you can hold against a requirement. It produces behavior. Behavior can be right in a thousand different ways and wrong in a thousand others, and you have written requirements for maybe ten of them. Try to express "drives well" as something you can measure. You can write "keeps the lane" and "holds the distance", and a system can pass every one of those and still do something in a crowded intersection that no requirement anticipated. The task has become too open to have an answer key.

Parts of this are hard in the old, familiar way. Guaranteeing redundancy in the computation, proving timing, bounding what the hardware must handle. Difficult, but difficult like engineering has always been difficult.

The other part is new. The behavior itself no longer decomposes into things you can assert. There is no list of checks that adds up to "drives well", and whatever list you write, the system's actual behavior is larger than it. Better test coverage does not close that gap. The gap is a property of the system.

And this is not a car story. It happens every time a system becomes capable enough to solve open problems, which is exactly where the rest of AI is heading. A coding agent that restructures a codebase is not producing an output you can diff against a correct answer. There is no correct answer. There are a thousand acceptable restructurings and a million subtle ways to be wrong, and your test suite covers a fraction of either. The same goes for the agent that answers your customers, drafts your contracts, plans your logistics.

Everyone assumes that better systems are easier to trust. In my experience it is the other way around. The dumb early systems were the ones you could verify to the bottom. The capable ones slip out of reach of your requirements, and they do it for a structural reason: capability means solving tasks too open to have an answer key, and no answer key means no proof.

The more it can do, the less you can prove.

So what do you do? You start by accepting the fact itself. You will never fully verify a capable system. Then you stop promising "it never fails" and build, seriously and expensively, for something else instead: detecting when it fails, limiting what a failure can touch, and being able to step in when it happens.

A safety engineer will now object that this is what people say when they have given up. That "limiting harm" is the white flag of someone who could not prevent it. I say this with respect, because I have been in those rooms, and the objection deserves a straight answer.

The answer is that there are two versions of accepting this truth, and they have nothing in common. The sloppy version never measures anything, does not notice when the system degrades, and could not intervene if it did. The disciplined version measures constantly, notices immediately, and can stop the system today. What separates them is not attitude. It is three things you either built or did not build.

The first is containment. If you cannot prove the system is right, the consequence of it being wrong has to be survivable. Actions that can be reversed. Scope that is bounded. A blast radius you decided in advance instead of discovering afterward. And a human on the decisions that cannot be taken back. Not on every decision. On the irreversible ones.

The second is observability, and I would argue it is the real dividing line for what you may deploy at all. Building to detect harm presupposes that you can detect it. A system whose behavior you cannot observe, running where you cannot intervene, is the one thing you must never release, no matter how good it seems in testing. I cannot know that a deployed system is correct. I can know what it did. That second kind of knowledge is available, and it is the one everything else rests on.

The third is the hardest to accept. Take the hardest case there is. The self-driving car. Maximally irreversible. Impossible to fully verify, for exactly the reasons above. Should it therefore not exist?

Look at what we require of the alternative. The human driver was never verified. We hand two tons of steel to people who are tired, distracted, angry, sometimes drunk, on the strength of one driving test taken decades ago. Nobody demands a proof of correctness from them. We accept them because, statistically, they mostly work. From the machine we demand perfection before it may touch the road, and we call that safety. It is not safety. It is a double standard, and it turns deadly the moment it blocks a system that is already better than we are.

So at the irreversible edge, the bar is not perfection. Perfection is not available. The bar is honestly measured superiority over the actual alternative. Not claimed once in a press release. Measured, continuously, in operation, with the willingness to see the day the answer changes.

Two warnings before anyone runs too far with that, because the car is a seductive example.

First, the car has something most systems do not: a measurable baseline. Decades of accident statistics to be better than. Your coding agent has no such number. Your credit model has a worse one than you think. "Better than the human" is only an argument where the human can actually be measured. Where there is no baseline, the phrase is not a measurement. It is a feeling, and it belongs in a brochure, not in a safety case.

Second, machines fail differently than we do. Humans fail one at a time, independently, each in our own way. A model fails correlated: every copy, the same error, the same rare input, the same second, across the entire fleet. A human who is bad on ice is a local problem. A model that is bad on ice is a national event. "Safer on average" can be entirely true and still hide a tail risk larger than any human population could produce.

There is a third thing, and it is not an engineering point. Statistically better is true in aggregate and cold in the particular. Somewhere there is a person the machine failed whom a human would have saved, and the statistics are no comfort to them. That does not change the math. But anyone who builds these systems should be able to hold that thought without flinching, because it is part of what the math costs.

Where I land is this. We will never fully verify a capable system, and demanding it is not safety. It is holding the machine to a standard we never held people to, and that double standard costs lives the moment it blocks something that is already better. But "already better" must never be a brochure sentence. It is a measurement you keep making, against the real alternative, with honesty about where you lack a baseline and where the failures correlate in ways an average conceals.

You are not trading verification away for faith. You are trading a guarantee you cannot give, "it never fails", for one you must be able to give: I measure that it is better. I will see when that stops being true. And I have built things so that I can step in when it does.

It is a much harder promise. It is also the only one of the two you can actually keep.