Every consequential system works from a representation of the people it decides about. A risk model represents an applicant. A triage model represents a patient. A fraud model represents a claimant. The representation is the system’s entire access to the person: it never meets them; it meets the data that stands in for them.
The framework’s ground rule about that arrangement is a short one:
A model of a person is not the person, and it never acquires automatic authority to extinguish the person’s standing against it.
Non-substitutability
The natural hope of machine learning applied to people is eventual substitution: a model good enough that the person’s own testimony, presence, and objection become unnecessary overhead. The rule refuses the inference. Accuracy is evidence about how the model will behave. It is not evidence about whether the person still matters, because the person is not a source of signals to be modeled but the party who will carry the consequences and whose standing is the thing no representation can stand in for.
This is why standing is inalienable rather than merely generous. If the subject of a decision could be replaced by a sufficiently detailed model of themselves, then challenging the decision would be a quality-control step, and a good enough model could close it. But the person is the one the decision happens to, and the mismatch between a life and any representation of it is exactly what their challenge is evidence about. A system that removes the person’s route into itself has not become more accurate; it has become less exposed to the one party positioned to know when it is wrong.
Safe incompleteness
The intuitive assumption is adversarial: better representation reduces the space where challenges matter, so the two trade off. Better representation and a preserved right to challenge compound instead, because challenge is the mechanism by which the representation’s errors are discovered at all.
The error this opposes is closure without remainder: the technocratic assumption that whatever the formal system does not represent has thereby been resolved. Under closure, every mismatch between the model and the person is read as user error, anomalous input, or noise. The system hears everything and learns nothing. This is absorption: each exception is handled, and none reaches the rule that produced it.
Better representation plus a preserved right to challenge, not better representation as a substitute for it.
A system designed around the expectation of its own incompleteness is not a lesser version of a system that expects completeness. It is the kind whose errors have somewhere to arrive.
Confidence does not convert into jurisdiction
Ordinary system design couples confidence to automation: the more confident the model, the less review. Confidence answers one question: is the inference probably correct? Jurisdiction answers another: may this institution act on this person on this basis? The first is about the model. The second is about the arrangement, and the non-conversion principle applies to it in full: high confidence is evidence that a grant of authority might be justified, never the grant itself.
The distinction matters most where confidence is highest, because that is where review gets optimized away. A system that is right about a person ninety-nine times out of a hundred has not thereby acquired the right to be final about the hundredth, and the institutional question — who may decide, for whom, until when — is untouched by the answer to the statistical one.
Who the system serves and who it decides about
The rule also exposes a confusion in user-centered design: the user is not always the person the system governs. In any consequential deployment these roles are held by different parties:
- The operator uses the system: the caseworker, the officer, the clinician at the desk.
- The beneficiary captures its value: the institution, the platform, the budget line.
- The subject is decided about: the applicant, the patient, the claimant.
- The residual custodian carries what the design could not represent: whoever absorbs the exceptions, which is often the subject and often also the operator.
Design that optimizes for the user can still be hostile to the subject. The insurer’s claims system can be excellent for the adjuster and ruinous for the claimant, and nothing in the user’s experience will show it, because the subject’s experience is not the user’s. Robodebt shows the split. The scheme took debt notices from about 20,000 a year to 20,000 a week, which is the beneficiary’s metric. The people it decided about were asked to produce payslips, often for years past, to disprove debts that income averaging could not establish. Naming the roles is what makes the question answerable: whose experience is this system designed around, and who is carrying what it cannot represent?
Finite capacity and the subject
The subject is not only a holder of rights. They are a person with limited time, attention, working memory, and emotional stamina.
When an administrative or algorithmic system leaves its gaps for the subject to bridge, it treats their adaptive capacity as unlimited. A welfare system that requires thirty hours of document retrieval, an appeal process that demands legal literacy, or a patient portal that requires constant self-advocacy turns procedural standing into an obstacle course.
Disability justice and feminist social reproduction theory both make the point this rests on: requiring exceptional individual endurance to get through ordinary institutional encounters is extraction. A model of a person that assumes unlimited capacity is as wrong as one that assumes unlimited compliance. An institution cannot design on the assumption that the subject has the time, energy, and resources of a full-time administrator assigned to the institution’s own errors.
Standing against, not control over
“Should a human be in the loop” is a question about control over the machine, and an arrangement can be full of human control and still be closed to the people it governs. The safeguard the framework cares about is standing against the institution, and it belongs to whoever the decision falls on. The important human may be entirely outside the operating loop: the one who can initiate review, compel explanation, halt execution, reverse the outcome, alter the rule, and challenge whether the institution should be deciding this at all.
Control without standing is administration. Standing is what makes a consequential system answerable to the people it can never fully represent. That is the positive form of the discipline: building capable systems that remain answerable to them. If you cannot represent a person adequately, do not pretend your representation exhausts what is relevant about them.
What the laws take from this
Law III: evidence and authority must remain coupled. A model’s confidence is evidence about the model. High confidence stays an input to a grant and never becomes the grant.
Law VII: error-bearing parties require standing proportional to exposure. The subject of the decision holds standing however accurate the representation becomes.
Law XII: no system may erase the conditions of its own contestability. A representation that replaced the person’s route into the system would erase one.
The narrow case for all three needs no view about persons. A system that the person it decides about cannot contradict has no way to learn when its representation is wrong. High confidence is the worst time to remove that channel: errors are rarest then, so each one that gets through is least likely to be caught.