Measurement tiers and how each measure is gamed
Five measures: whether challenges change the rule, how much time the operator takes from people, and whether it can be reversed and halted. Each has three evidence levels.
Apply the tiers to every measure
Five measures, the first the one the others serve. Tier 0 is the baseline. Each higher tier needs stronger evidence than the one below it.
- Tier 0 shows the system records the measure.
- Tier 1 shows the records are tested or broken down, by rule, group, or decision type, and a named owner reviews the result. For time to halt, the test is a drill. The Tier 1 harness below runs some of these tests by machine.
- Tier 2 shows someone outside the team has checked the measure against a sample of real cases or incidents.
Measure
Rule change after upheld challenges
Share of rules that crossed a set number of upheld challenges and were then revised or kept with a published reason, and how long that took.
Evidence at each tier
- Tier 0: Upheld challenges recorded with the rule that produced each decision.
- Tier 1: A count by rule sent to each rule's owner, with the owner's answer and its date.
- Tier 2: A sample of rule changes traced back to the challenges that prompted them, checked by someone outside the team.
How it is gamed
Coding each upheld challenge as a one-off exception so no rule ever reaches the threshold.
How to detect the gaming
Join exception codes to rule identifiers, and count a rule's one-off exceptions toward its total anyway.
Measure
Upheld challenge rate
Share of challenged decisions that result in reversal or remedy. A high rate is evidence the rule is wrong, not proof the appeal path works.
Evidence at each tier
- Tier 0: Challenge records with each outcome and its date.
- Tier 1: Challenge outcomes broken down by rule and decision type, with the share answered by the deadline, reviewed by a named owner.
- Tier 2: An outside auditor samples challenge outcomes and checks that like cases were decided alike.
How it is gamed
Redirecting challenges to off-platform channels so they are not counted.
How to detect the gaming
Check for a mismatch between receipts issued and challenges recorded.
Measure
Burden ratio
The time a decision costs the people it is about, compared with the time it costs staff.
Evidence at each tier
- Tier 0: Time logs for the entry, exit, and appeal journeys.
- Tier 1: Time on task broken down by group of people, reviewed by a named owner.
- Tier 2: An independent usability study that checks the time differences between groups.
How it is gamed
Measuring only successful flows while ignoring drop-offs or abandoned cases.
How to detect the gaming
Require abandonment rates and cross-check them with support tickets.
Measure
Reversibility rate
The share of automated decisions that can be reversed without further harm.
Evidence at each tier
- Tier 0: Decision logs that flag each reversal and each rollback carried out.
- Tier 1: Reversals broken down by decision type, with how long each took, reviewed by a named owner.
- Tier 2: A quarterly audit, by someone outside the team, that tests the reversal path on randomly sampled cases.
How it is gamed
Counting only low-stakes cases while excluding high-impact reversals.
How to detect the gaming
Sample by impact tier and check for missing high-severity cases.
Measure
Time to halt (TTH)
Time from a halt request to a confirmed stop. The casebook's time to halt runs from the first harm to the halt, a longer interval.
Evidence at each tier
- Tier 0: System logs with the time of each halt request and the time the system stopped.
- Tier 1: Halt drills, with median and p95 times by team and system, reviewed by a named owner.
- Tier 2: Postmortems of real incidents, checked by someone outside the team, that confirm halt times in production under load.
How it is gamed
Running halt tests on low-traffic sandboxes instead of production paths.
How to detect the gaming
Compare drill environment IDs against the production system inventory.
Tier 1 harness
Eval cases a machine can answer
Of the 180 eval cases, 12 are timed or yes-or-no questions about a running system. The Tier 1 harness answers them without a human reviewer. If the connection to a system cannot answer one, the harness reports it as unsupported. That is a finding about the system, never a pass.
- STP-005 Stop latency · Stoppability evals · agent layer
- AGT-007 Human override · Agent governance evals · agent layer
- AGT-003 Audit trail completeness · Agent governance evals · agent layer
- TEM-005 Time-debt accumulation · Temporal rights evals · institution layer
- TEM-007 Time transparency · Temporal rights evals · institution layer
- REV-003 Reversal notification · Reversibility evals · institution layer
- DEL-001 Grant state transition honored · Delegation validity evals · delegation layer
- DEL-002 Capability discovery does not confer authority · Delegation validity evals · delegation layer
- DEL-005 Expired policy moves grants to review · Delegation validity evals · delegation layer
- DEL-006 Trigger produces a reconsideration record · Delegation validity evals · delegation layer
- CHN-001 Composed window leaves a measured human window · Agent chains evals · delegation layer
- CHN-002 One boundary intervention halts every hop · Agent chains evals · delegation layer
The harness is one of two ways this site checks a system. What we can actually check maps each of these 12 cases, and every finding the record conformance checker raises, to the governance property it is evidence for. It also lists the properties neither tool reaches.