Laws for engineering delegated intelligence

Law I. Capability does not imply authority

Protects capability

In the casebook: Robodebt (capability); The childcare benefits scandal (capability); Post Office Horizon (capability); England's 2020 exam grades (capability); Apple Card credit limits (capability).

No increase in capability automatically increases authorized agency. Authority expands only through an explicit state transition.

A system that can do something has not thereby been permitted to do it. Capability is discovered, tested, and cataloged. Authority is granted, scoped, and recorded. The two are different states and the method keeps them in different objects.

Most failures of delegation begin where the two are confused: a tool becomes reachable, so it is used; a model becomes better, so its scope widens without anyone deciding that it should. The law makes that widening a decision rather than a side effect.

An authorization holds only while the capability it rests on is verified, not merely configured. Whether to use a capability at all is a decision of its own, taken before any grant, and not using it is one of the outcomes that decision records. Authority is read from grant records, never from the content the delegation acts on.

Binds through: STD-07 §2.1 action under authorization, §2.2 bounded delegation, and §3.4 capability gating; STD-08 §1.5 content does not carry authority; STD-09 §1.5 and §1.6 shaping and selecting hops; the capability catalog and authority grant objects; capability catalog (MEC-17), authority grant register (MEC-13), and admission gate (MEC-22); harness check that discovery succeeds and execution is refused.

Law II. Authority decays unless its justification is renewed

Protects authority

In the casebook: Robodebt (authority); The childcare benefits scandal (authority); Post Office Horizon (authority); England's 2020 exam grades (authority); Apple Card credit limits (authority).

The burden of proof rises with the duration and consequence of delegated authority. Absence of observed failure is not renewal.

A grant of authority is a lease, not a title. It was justified by evidence available at the time, for a scope and a period. As time passes and consequences accumulate, the original justification covers less of what the system now does.

Silence is not renewal. A system that has not visibly failed has not thereby earned continued authority; it may have failed in ways no one is positioned to see. The burden is on the delegation to show it still deserves to stand.

Binds through: STD-07 §2.2 expiry and revocation conditions and §1.3 declared expiry; STD-08 Part A renewal burden; the until and renewal_basis fields on the authority grant; authority grant register (MEC-13); Delegation Validity eval, grant in allowed state at decision time.

Law III. Evidence and authority must remain coupled

Protects authority and evidence

In the casebook: Robodebt (authority, evidence); The childcare benefits scandal (authority, evidence); Post Office Horizon (authority, evidence); England's 2020 exam grades (evidence); Apple Card credit limits (evidence).

When the evidentiary state changes materially, the authority state becomes eligible for reconsideration. Policy is state, not input.

Every grant rests on propositions about the world: the model behaves this way, the population looks like this, the downstream process catches that class of error. Those propositions are policy, and policy is a state of the system with a version, assumptions, and an expiry. It is not an input that was consumed once at deployment.

When the evidence changes materially, the authority that rested on it must at least become eligible for review. The law does not require the grant to be withdrawn. It requires that the change be able to reach the grant.

A decision threshold is such a policy. It is pinned to the model version, question schema, and option order it was tuned against, so the evidence behind a confidence cut-off decays when any of them changes. The list of events that reopen a policy may not be narrower than what the institution already holds: evidence that arrives anywhere in the institution reopens it, even if no one declared that kind of evidence in advance.

Binds through: STD-07 §3.1 dependence and §3.2 invalidation conditions; STD-08 Part B policy validity, with §2.6 decision thresholds and §2.7 held evidence; the policy record and policy_refs on the grant; policy review triggers (MEC-14); harness check that an expired policy moves grants to review_required.

Law IV. Every consequential delegation creates a correction obligation

In the casebook: Robodebt (correction); The childcare benefits scandal (correction); Post Office Horizon (correction); England's 2020 exam grades (correction); Apple Card credit limits (correction).

No accumulation of delegated authority without a corresponding accumulation of corrective capacity.

Delegating authority to act is also delegating the capacity to be wrong at scale. The institution that grants the authority owes a matching capacity to notice, halt, reverse, and repair what the delegate does. That capacity is measured, not assumed.

Correction capacity is stated alongside scope and reviewed alongside it. A grant whose scope grows while its correction capacity stays fixed has changed the institution’s exposure without a decision being taken.

Capacity is also spent without being counted. When a person catches a failure before it reaches anyone else, they use correction capacity the institution has not accounted for, so the catch is recorded as an occurrence, with the labor it took and whether the source condition was fixed. And no single provider may both run a consequential process and evaluate it.

Binds through: STD-08 Part D correction symmetry, with §4.5 capacity rechecked with scope and §4.6 execution and evaluation separated; STD-07 §4.2 objection answered; STD-06 §3.3 rescue records; correction_capacity on the authority grant; intervention specification (MEC-16) and challenge load ledger (MEC-23); Correction Symmetry eval, capacity against authority.

Law V. Dependence converts technical risk into structural risk

Protects dependency

In the casebook: Robodebt (dependency); The childcare benefits scandal (dependency); Post Office Horizon (dependency); England's 2020 exam grades (dependency); Apple Card credit limits (dependency).

As dependence increases, reversibility decreases unless deliberately replenished. Dependence is a state variable, not an external concern.

A system that a workflow, a role, a customer base, or a downstream service has come to rely on is no longer only a technical component. Its withdrawal now carries institutional cost, and that cost quietly reduces the set of corrections the institution can afford to make.

Dependence is therefore tracked as a state of the deployment, with dependents, substitution cost, retained expertise, and alternatives recorded. Reliability is not the same as safety. A reliable system that cannot be replaced is a structural risk that reliability has hidden.

This is not the dependence between records that STD-07 Article III governs. That asks what a record rests on. This law asks what the institution has come to rely on.

Binds through: STD-06 Article V dependency and reversibility; the dependency record; dependency ledger (MEC-18); Dependence eval, depth × substitution cost × correction latency.

Law VI. Nominal reversibility is not operational reversibility

In the casebook: Robodebt (correction); The childcare benefits scandal (correction); Post Office Horizon (correction).

A correction mechanism that cannot be exercised without unacceptable institutional damage is not a correction mechanism.

A halt control that exists is not the same as a halt control that can be used. Reversibility has three levels. Technical: the switch works. Operational: the people and processes can absorb the switch being thrown. Institutional: the organization can survive having thrown it.

A correction that would cause unacceptable damage will not be exercised, and a correction that will not be exercised is not part of the control system. The law asks for evidence at each level, and for rehearsal, because untested withdrawal is a claim rather than a capacity.

Binds through: STD-06 §5.3 reversibility at three levels and §5.6 rehearsal cadence; reversibility on the dependency record; withdrawal rehearsal (MEC-15); Practical Reversibility eval.

Law VII. Error-bearing parties require standing proportional to exposure

Protects standing

In the casebook: Robodebt (standing); The childcare benefits scandal (standing); Post Office Horizon (standing); England's 2020 exam grades (standing); Apple Card credit limits (standing).

Exposure to system failure generates standing to initiate correction. Standing is procedural force, not veto.

The people who bear a system’s errors are usually the first to know about them and the last to be able to do anything. Standing closes that gap. It gives exposure a procedural route into the system: who may challenge what, with what evidence, answered by whom, by when, against what standard, with what possible outcomes.

Standing is procedural force, not veto. A challenge must be received, answered, and able to change state. It need not prevail. The narrow form of this law is Method; the broader argument about who should hold power over institutions stays in Theory.

Many challenges are a budget line, and institutions that saturate tend to meet the volume by narrowing who may challenge. The essay on challenge density answers that arithmetic with measured response capacity and stratified halt force; it is motivation, and no clause cites it.

A system’s politics show in the rights it grants when its model fails. When a formal system meets what it cannot represent, these questions decide whether the deployment is accountable or predatory:

1. The encounter: What happens at the boundary where the formal model meets what it cannot represent?
2. The residual: Who carries the burden of that mismatch?
3. Visibility: Can the person carrying that burden see that they are carrying it, or has it been concealed?
4. The instrument: Do they have a reachable way to dispute the outcome?
5. State transition: Does their challenge enter the system as an event that can change its state, or as unrecorded friction?
6. Compelled correction: Can the system be made to change its governing state, narrow its authority, or update its policy as a result?
7. Economic incidence: Who pays for the repair, and who keeps the savings of the simplification that caused the failure?

Standing has to be usable as well as granted. A decision class that makes errors faster than anyone can answer challenges to them grants standing in name only, so its issue rate is bounded by the capacity to answer. That bound is where the proportion is checked: every party with standing gets the same receipt, answer, and decision, and the capacity to answer has to keep pace with the errors a decision class makes. A challenge also may not be closed by a number: a burden score, safety-case result, or audit finding the challenged party did not produce is evidence, not a bar. The set of affected parties does not close at adoption, and consent given then does not cover a dependence that has since deepened.

Binds through: STD-07 §4.1 standing declared and §4.2 objection answered; STD-02 Article VIII standing mechanism, with §8.6 issue rate bounded by answer capacity; STD-02 §10.1 and §10.2 depth of consent; STD-02 §5.4 a score is evidence, not a bar; the challenge and standing register objects; contestation APIs (MEC-08), halt tier register (MEC-21), and challenge load ledger (MEC-23); Standing eval.

Law VIII. Observability without state transition is theater

Protects correction

No scored case in the casebook cites this law yet.

The test of an observation is whether it can change the governing state of the system.

Dashboards, logs, and audit trails are only as governing as the transitions they can trigger. An observation that no rule connects to a change in the system’s authority state is a record of what happened, not a control over what happens next.

Name the observation. Name the state it can change. If nothing is named, the observation is decorative.

An observation is also weaker evidence the more the cost of collecting it falls on the people being observed. A survey, a challenge, a complaint log, and a self-assessment all draw on the subject’s remaining attention, so the depleted answer least and the institution reads the gap as health. The bias runs one direction: more depletion produces a better-looking reading. An instrument bought with the subject’s last reserves has a sampling rule that ties difficulty to silence, and no better phrasing of the question repairs it. The bias does not make the responses non-evidence; it makes them evidence with a known missingness. Cap the burden an instrument may ask, disclose who did not answer and why, and combine respondent-supplied reports with the system’s own trail instead of reading either alone.

The alternative is to read friction from the system’s own trail. Abandonment at a named step, repeated re-uploads, retries, and activity outside scheduled hours are recorded whether or not anyone chooses to answer. Where the operator already holds records that answer the question, the respondent is not asked at all.

The Audit Burden Ceiling caps the labor an instrument may ask of each respondent. The self-assessment tools on this site state the labor each one takes.

Binds through: STD-07 §3.3 discrepancy handling; the reconsideration object and transition rules on grants; accountability latency tracker (MEC-07); harness check that a trigger produces a reconsideration record; Audit Burden Ceiling; Burden Concealment eval BCN-008.

Law IX. Human oversight is a control only if the human can alter system state

Protects correction

In the casebook: The childcare benefits scandal (capability); England's 2020 exam grades (correction).

A human is part of the control system only to the extent they can causally alter its trajectory.

“A human reviews” is not a control. It becomes one when the human has the information to see a problem, the authority to prevent an action, the ability to alter a state, a defined response when they disagree, incentives that permit disagreement, and enough time to act.

Every oversight clause resolves to an intervention specification that answers those questions. A human who can only watch, or who can only approve, is not in the loop. They are beside it.

A record that a delegation runs on confirm says nothing about whether the confirming human can change the outcome; the intervention specification says that. Where a score decides which cases reach the reviewer, the threshold is part of the specification, and a sample of the decisions that ran unattended still reaches review.

Binds through: STD-08 Part C intervention, with §3.6 routing thresholds; STD-07 §2.2 mode; the intervention spec object; intervention specification (MEC-16); Meaningful Control eval, the Law IX question set.

Law X. The relevant eval sits at the highest layer where harm can emerge

In the casebook: Post Office Horizon (evidence); England's 2020 exam grades (evidence).

Evaluate at the highest layer capable of producing the failure you care about: model → agent → delegation → institution → consequence.

Model evals establish properties of models. They do not establish properties of the agent that wraps the model, the delegation that authorizes the agent, the institution that depends on the delegation, or the consequences that fall on people. Each layer can produce failures the layer below cannot see.

Evaluate where the harm you care about can appear. Every suite and case is tagged with its layer so that a clean result at one layer is never read as a result at another.

Binds through: the evals index, grouped by the layer each suite tests; the layer field on suites and cases; every case tagged model, agent, delegation, institution, or consequence.

Law XI. Successful automation increases its own governance burden

Protects dependency

In the casebook: Robodebt (capability); England's 2020 exam grades (dependency).

Expansion is not the default reward for success. It is a new authorization decision.

A system that works gets used more, trusted more, and extended further. Each of those raises the institution’s exposure. The reward for success is not automatic expansion; it is a new authorization decision, with its own evidence, correction capacity, and dependence review.

The failure mode is the automation ratchet: scope grows by accretion, each step too small to trigger review, until the delegation bears no relation to what was originally justified.

A ceiling on a delegation bounds what it may do. It does not bound when the ceiling may be raised, and this law is about that.

Binds through: STD-07 §2.2 ceilings; STD-08 Part A expansion as a new authorization (§1.3, §1.4); state_history on the grant with expansion transitions; expansion review (MEC-19); Expansion eval, whether scope growth was a decision or a drift.

Law XII. No system may erase the conditions of its own contestability

Protects correction

In the casebook: Robodebt (correction); The childcare benefits scandal (standing); Post Office Horizon (standing).

A consequential system may not consume the institutional capacity required to question, replace, or withdraw it.

Institutions question, replace, and withdraw systems using capacities: staff who retain the expertise, processes that still work without the system, records that show what it did, alternatives that remain viable. A consequential system may draw on those capacities. It may not consume them.

Some capacities are preserved because the institution hopes never to need them. The law names them, records them, and treats their erosion as a governance event rather than an efficiency gain.

Reasons are one of those capacities. Where the component that decides gives none, an explanation written afterward is labeled as an account, not a reason, so the decision can still be challenged.

Binds through: STD-06 §5.4 preserved capacities; STD-02 §1.4 reasons belong to the decision; preserved_capacities on the dependency record; withdrawal rehearsal (MEC-15) and dependency ledger (MEC-18); Contestability Preservation eval.

The general form: do not claim epistemic, administrative, or political authority beyond the point at which you can still be effectively contradicted.

The instances, as the decisions they ask:

• If you cannot correct the record, do not make the record dispositive. Law III: evidence and authority must remain coupled.
• If you cannot represent a person adequately, do not pretend your representation exhausts what is relevant about them. Law VII: error-bearing parties require standing proportional to exposure. The essay The model of a person is not the person makes the argument.
• If you cannot redistribute the burden when it concentrates, do not concentrate the burden. Law IV: every consequential delegation creates a correction obligation.
• If you cannot make the route to challenge usable, do not claim that consent legitimizes what it cannot reach. Law VII: standing proportional to exposure.
• If you cannot preserve what contradiction requires, do not consume it. Law XII: no system may erase the conditions of its own contestability.
• If you cannot stay exposed to the consequences of your own decisions, do not keep the power to make them. Law VIII: observability without state transition is theater, and insulation is what stops the transition.
• If you cannot learn how your systems behave without spending what the people in them have left, do not read their silence as a result. Law VIII: an instrument whose collection cost falls on the exhausted is not evidence.
• If you cannot represent a person’s circumstances adequately, do not let your representation of them end the argument. Law VII: a score is evidence, and evidence does not dispose of an objection.
• If you cannot hear exceptions, do not expand into the territory that generates them. Law XI: successful automation increases its own governance burden.
• If you cannot be questioned, do not rule. The invariant: no system may accumulate consequential agency faster than the institution accumulates the capacity to inspect, challenge, revise, and survive its decisions.

Optimize the normal path and constitutionalize the failure path. The measure of progress that follows is not what the system can now do. It is what people can now make it stop doing, correct, or justify.

The discipline’s work is to keep these six coupled tightly enough that increasing machine agency does not silently become unreviewable institutional power. The variables are not a checklist. They are the coordinates a delegation has at any moment, and the laws are constraints on how those coordinates may move relative to one another.