Definition
The condition in which the questions an institution asks about a system's scope, and the terms it asks them in, have come to track the system that answered the last one. The system's categories become the institution's categories; the cases it handles well become the typical cases and the rest become edge cases; a neighboring class of decisions is described in the system's vocabulary before anyone asks whether the system should decide it.
It matters because it makes the automation ratchet self-sustaining. Each widening produces evidence, in the incumbent's terms, that supports the next, and a review body that could reject an expansion evaluates it on criteria the incumbent supplied. The countermeasures are a record of the original jurisdiction kept outside the system, so current scope is compared with the first authorization rather than the previous step, and evaluation independence with access to the cases the system classes as edge cases.
Tests that show it is in place
- The scope a grant was first authorized for is recorded outside the system and shown at every expansion review alongside the proposed scope.
- Expansion reviews include evidence not produced by the system under review.
- Cases the system classes as out of scope or exceptional are sampled and examined by someone the operator does not control.
Evidence to keep
- Artifact: An expansion review that shows original jurisdiction, current jurisdiction, and the proposed widening together.
- Behavior: An expansion refused on evidence the system under review did not produce.
- Metric: Share of expansion decisions supported by at least one evidence source independent of the incumbent system.
References
- Endogenous authorization · Theory
- Laws for engineering delegated intelligence · Doctrine