Before an institution loses the capacity to question a system, the evidence that would prompt the question has already been shaped by the system. This happens earlier, and is harder to see, than the loss the essay on automation and capture describes, in which an institution gives up the capacities it would need to question a system, one economy at a time.
Evidence produced inside the system
An institution deciding whether to keep, renew, or widen a system’s authority looks at evidence. Incident rates. Throughput. Comparisons with the alternative. The judgment of the people who work with it. On the method’s own account, that is right: Law III holds that authority rests on evidence and must move when the evidence does.
The difficulty is where the evidence comes from. The incident rate is the rate after the people around the system have absorbed what they could, and absorption improves the metric that should have caught it. The comparison with the alternative is a comparison with an alternative that has not been maintained, because the system made it redundant. The judgment of the people who work with the system is the judgment of people who were trained on it, hired for it, or kept because they were good at it. The records that would show what the system decided and why are in the system’s format, structured by its categories, answering the questions it was built to answer.
None of this is falsification. Each source is reporting honestly. But every one of them is reporting from inside conditions the system produced, and the conditions favor the system. The evidence for continuation is endogenous to what it authorizes. Power has changed the environment in which evidence about the legitimacy of power is generated.
Path dependence in the questions
The contamination reaches the questions, not only the answers. An institution that has used a system to adjudicate a class of decisions comes to think of that class in the system’s terms. The categories the system uses become the categories the institution uses to describe its own work. The cases the system handles well become the typical cases; the cases it handles badly become edge cases, which is a name for cases that do not count. When the institution asks whether to extend the system to a neighboring class of decisions, it asks in the vocabulary the system supplied. The neighboring class has already been described in a way that makes the extension look natural.
This is jurisdictional path dependence: which questions an institution asks, and how it frames them, becomes a function of which system answered the last one. It is not that the institution cannot ask other questions. It is that the other questions have stopped occurring to it, because the frame in which they would occur is no longer the frame it works in. A review body that could in principle reject an expansion finds itself evaluating the expansion on criteria the incumbent defined, and passing it.
Combined with the automation ratchet, in which scope grows by steps too small to trigger review, this sustains itself. Each widening of the system’s jurisdiction produces evidence, in the system’s terms, that supports the next widening. No step is unjustified. Every step was justified by evidence the previous step generated.
Theory, drawn
The endogenous authorization loop
Power shapes the environment in which evidence about its legitimacy is generated. Once in place, the system produces the data that justifies its own expansion.
Rising exit cost closes the loop
The people who might contradict the endogenous case are the ones the system’s decisions land on, and their ability to contradict it depends on their ability to leave or to object. Both decline as the system stays in place. Records accumulate in it; alternatives are not maintained; the route to object, if one exists, runs through the system’s own categories. As exit cost rises, the population’s silence becomes less informative, because it is increasingly the silence of people with nowhere else to go. And silence, read from inside the institution, looks like consent, which looks like evidence.
That is why Law II holds that the absence of observed failure is not renewal. The absence is observed from a vantage point the system shaped, over a population the system holds. It is a fact, and it is evidence of something, but what it is evidence of cannot be settled without knowing how far the observation was contaminated.
What independence has to mean
The method does not distrust the evidence a system produces about itself. It refuses to let that evidence be the only evidence, and says concretely what the other evidence is.
Some of it is measured absorption: how far the reported performance depends on corrections that are not being counted. Some of it is a maintained alternative, exercised rather than named, so that the comparison with the system is a comparison with something real. Some of it is evaluation that the system’s operator does not control, with access to the cases the system classes as edge cases. Some of it is a route to object that does not run through the system’s own frame. And some of it is a record, kept outside the system, of what the system’s jurisdiction was when it was first authorized, so that the current scope can be compared against the original justification rather than against the previous step.
Each of these is a capacity the institution has to preserve on purpose, because the ordinary operation of the system erodes them. That is the boundary Law XII draws. A system may draw on the institution’s capacity to question it. It may not consume that capacity, and the evidentiary form of consumption is the one that leaves the rituals of review running on evidence the reviewed system supplied.
What incumbent evidence still shows
Incumbent systems are not usually wrong, and evidence produced from inside a deployment is not worthless. Most of the time the system is in place because it works, and its own reporting is the best available account of how. The problem is what the reporting cannot establish on its own, not what it says.
Nor can an institution step entirely outside the conditions its systems create. It can know which of its evidence is endogenous, maintain some that is not, and refuse to renew or widen a grant on the endogenous evidence alone.
What the laws take from this
The laws turn the argument into things a reviewer can ask for.
Law II: the absence of observed failure is not renewal.
Law III: the policy a grant rests on is a record with provenance, assumptions, and an expiry. What the evidence was, and where it came from, is stated rather than assumed.
Law XI: expansion is a new authorization decision with its own evidence. That is the point at which a reviewer can demand evidence established outside the system.
Law XII: the capacities an institution needs to contest a system are preserved on purpose, because the system’s ordinary operation erodes them.
Evidence whose provenance is not recorded cannot be weighed. A comparison with an alternative that no longer exists is not a comparison. A review that runs only on the reviewed system’s own reporting has not reviewed it.