Every consequential system works from a representation of the people it decides about. A risk model represents an applicant. A triage model represents a patient. A fraud model represents a claimant. The representation is the system’s entire access to the person: it never meets them, it meets the data that stands in for them.
The framework’s ground rule about that arrangement is a short one:
A model of a person is not the person, and it never acquires automatic authority to extinguish the person’s standing against it.
Everything else in this essay is a consequence of taking that rule seriously.
Non-substitutability
The natural hope of machine learning applied to people is eventual substitution: a model good enough that the person’s own testimony, presence, and objection become unnecessary overhead. The rule refuses the inference. Accuracy is evidence about how the model will behave. It is not evidence about whether the person still matters, because the person is not a source of signals to be modeled but the party who will carry the consequences and whose standing is the thing no representation can stand in for.
This is why standing is inalienable rather than merely generous. If the subject of a decision could be replaced by a sufficiently detailed model of themselves, then contestation would be a quality-control step, and a good enough model could close it. But the person is the one the decision happens to, and the mismatch between a life and any representation of it is exactly what their challenge is evidence about. A system that removes the person’s route into itself has not become more accurate; it has become less exposed to the one party positioned to know when it is wrong.
Safe incompleteness
The companion claim is about what better modeling buys. The intuitive assumption is adversarial: better representation reduces the space where contestation matters, so the two trade off. The framework’s claim is the opposite. Better representation and preserved contestation compound, because contestation is the mechanism by which the representation’s errors are discovered at all.
The error this opposes is closure without remainder: the technocratic assumption that whatever the formal system does not represent has thereby been resolved. Under closure, every mismatch between the model and the person is read as user error, anomalous input, or noise. The system hears everything and learns nothing, which is the absorption mechanism: exception after exception handled and no exception ever reaching the machinery.
Better representation plus preserved contestation, not better representation as a substitute for it.
A system designed around the expectation of its own incompleteness is not a lesser version of a system that expects completeness. It is the only kind that can survive contact with the world, because it is the only kind whose errors have somewhere to arrive.
Confidence does not convert
A further consequence concerns uncertainty. Ordinary system design couples confidence to automation: the more confident the model, the less review. Epistemic confidence answers one question — is the inference probably correct — while jurisdiction answers another — may this institution act on this person on this basis. The first is about the model. The second is about the arrangement, and the non-conversion principle applies to it in full: a high posterior is evidence that a grant decision might be justified, never the grant itself.
The distinction matters most where confidence is highest, because that is where review gets optimized away. A system that is right about a person ninety-nine times out of a hundred has not thereby acquired the right to be final about the hundredth, and the institutional question — who may decide, for whom, until when — is untouched by the answer to the statistical one.
The four roles
The rule also exposes a confusion in user-centered design: the user is not always the person the system governs. Four roles recur in any consequential deployment, and they are held by different parties:
- The operator uses the system: the caseworker, the officer, the clinician at the desk.
- The beneficiary captures its value: the institution, the platform, the budget line.
- The subject is decided about: the applicant, the patient, the claimant.
- The residual custodian carries what the design could not represent: whoever absorbs the exceptions, which is often the subject and often also the operator.
Design that optimizes for the user can still be hostile to the subject. The insurer’s claims system can be excellent for the adjuster and ruinous for the claimant, and nothing in the user’s experience will show it, because the subject’s experience is not the user’s. The casebook’s failures are largely failures of this confusion: systems whose subjects had standing in name only, while the operator’s queue and the beneficiary’s metric both looked healthy. Naming the roles is what makes the question answerable: whose experience is this system designed around, and who is carrying what it cannot represent?
Standing against, not control over
The last consequence reframes the familiar oversight question. “Should a human be in the loop” is a question about control over the machine, and an arrangement can be full of human control and still be closed to the people it governs. The safeguard the framework cares about is standing against the institution, and it belongs to whoever the decision falls on. The important human may be entirely outside the operating loop: the one who can initiate review, compel explanation, stop execution, reverse the outcome, alter the rule, and challenge whether the institution should be deciding this at all.
Control without standing is administration. Standing is what makes a consequential system answerable to the people it can never fully represent, which is the positive form of the whole discipline: the practice of building powerful systems that remain answerable to them. The epistemic injunction follows from the ground rule and closes the argument: if you cannot represent a person adequately, do not pretend your representation exhausts what is relevant about them.
What the laws take from this
The method does not require a reader to accept that a model of a person never acquires authority over the person. It takes three constraints. From Law III it takes that a model’s confidence is evidence about the model and never the decision, which is why a high posterior stays an input to a grant rather than the grant. From Law VII it takes that error-bearing parties require standing proportional to exposure, and that the subject of the decision holds standing regardless of how accurate the representation becomes. From Law XII it takes that no system may erase the conditions of its own contestability, which is what a representation that replaces the person’s route into the system would be doing. An engineer who rejects the essay can adopt all three on narrower grounds: a system that cannot be contradicted by the person it decides about has no way to learn when its representation is wrong, and confidence is the worst moment to remove that channel, because it is when the errors are rarest and the stakes are highest.