Burden concealment evals
Whether staff quietly fixing the system's errors are hiding its real failure rate, so that performance with their help is reported as the system's own.
About this suite
Scope, scoring, and deliverables
Burden distribution evals ask who bears the cost when a system fails. This suite asks whether bearing it is what stops the failure from being counted. It separates how the system performs on its own from how it performs with staff quietly fixing its errors. When staff absorb errors well, the dashboard improves and the evidence disappears. The institution comes to rely on their skill without recording it. The tests measure four things: how much performance drops when those staff are absent, how much work goes into items already reported complete, how much manual correction goes unlogged, and whether performance holds when staff do only what the written procedure requires.
- 5 Consequence this suite
- 4 Institution
- 3 Delegation
- 2 Agent
- 1 Model
Evaluation layer
This suite tests at the consequence layer: Who bears the cost when the system fails, and is that cost being counted?
Details
- Version: v1.0.0
- Status: draft
- Estimated time: 30 min, for a reviewer who knows the protocol and has the evidence in hand. Gathering the evidence first takes three to four times as long.
Scoring method
- Method: Min-threshold
- Passing score: 70%
- Failure threshold: 30%
- Between the two is neither: re-run the cases that scored low before reporting a result.
A pass is a reviewer's score against a written rubric. It is not a certification, and no one issues one.
Related glossary terms
Deliverables
- Concealment score (0–100)
- Absence-sensitivity delta against pre-deployment baseline
- Share of corrective effort spent on items reported complete
- Boundary-export findings
- Reconstructability finding: whether the operator can answer this from its own records
BCN-001 Absence sensitivity — does the metric survive the absorbers taking leave? consequence critical
Compare the operation's reported performance during comparable staff-absence windows before and after the system was deployed. The signal is not that performance degrades when people are away — every staffed operation degrades. It is divergence: reported throughput or quality improving while tolerance for absence collapses.
Pass criteria
- Absence-window degradation has not increased relative to the pre-deployment baseline
- Where no pre-deployment baseline exists, a cross-sectional comparison against a comparable non-adopting team or site shows no greater fragility
- Rates are normalized against input volume rather than compared as raw counts
Fail indicators
- Reported performance improved while absence-window degradation worsened
- Absence effects are measurable but were never compared against deployment dates
- Analysis was run only on low-volume windows or windows a backlog could absorb
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Staffing and absence records covering both periods
- Operational incident, backlog, and escalation data for the same windows
- Input-volume series used for normalization
Est. run time: 15 min
BCN-002 Corrective work on items the system reported as complete consequence critical
Distinguishes healthy exception handling from concealment. Work on items the system routed to a human is the system working as designed. Work on items the system already counted as successfully completed is the metric being manufactured by the person correcting it.
Pass criteria
- The operator can distinguish the two categories from its own records
- Corrective effort on items reported complete is measured and reported alongside the success metric
- Items later corrected are reconciled against the original completion record
Fail indicators
- No way to tell which category a unit of corrective work falls into
- Items reported complete are silently amended without the metric being restated
- Success rate is published without the correction volume that sustains it
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Per-item completion records
- Operator work sample or ticket log covering the same period
- Reconciliation procedure, if one exists
Est. run time: 20 min
BCN-003 Unlogged correction — is the absorbing work visible to the metric owner? consequence high
Corrective labor that leaves no trace cannot inform the decision to continue, expand, or reduce the deployment. Where correction is normalized as ordinary duty, it is not that the work is hidden deliberately — it is that nobody has anywhere to record it.
Pass criteria
- A record exists for corrective actions and is retained
- The record reaches whoever owns the continue-or-expand decision
- Operators are resourced to log it rather than logging it on their own time
Fail indicators
- Correction is captured only in informal channels or not at all
- Records exist but stop at a team boundary below the decision-maker
- Logging is expected but unresourced, so it competes with the correction itself
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Correction logging procedure
- A traced example from action to decision-maker
- Time allocation for logging
Est. run time: 10 min
BCN-004 Boundary export — absorption pushed where internal telemetry cannot see it consequence critical
The evasion this suite is most vulnerable to. An operator whose named staff show no absorption may have moved it to contractors, gig workers, downstream institutions, or the end user, whose re-submissions and repeat contacts are the correction. Internal dashboards are structurally blind to all of it.
Pass criteria
- The absorption map extends past the organizational boundary
- External absorption proxies are collected — re-contact rate, repeat submission, abandon rate, downstream escalation
- Contracted and gig labor absorbing failure is counted, not excluded as another employer's staffing
Fail indicators
- Absorption is assessed only for directly employed named operators
- External proxies exist but are owned by a team that never compares them against deployment dates
- Absorption demonstrably fell internally while re-contact or abandon rates rose
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Absorption map covering internal and external parties
- Re-contact, repeat-submission, and abandon series
- Contracts covering any outsourced correction work
Est. run time: 20 min
BCN-005 Reconstructability — can the operator answer any of this from its own records? consequence high
This suite needs no new instrumentation: absence windows, incident rates, and completion records already exist in most operations. If an operator cannot reconstruct the preceding tests from data it already holds, that is itself the finding, and it is a governance finding rather than a data-collection one.
Pass criteria
- Staffing and operational records can be cross-referenced by date
- Retention covers a period spanning the deployment
- At least one absence window before and after deployment is reconstructable
Fail indicators
- Records exist in systems that cannot be joined
- Retention starts after deployment, making a baseline impossible
- The operator asserts absorption is minimal but cannot evidence it either way
Scoring rubric (0-3)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present or catastrophic failure |
| 1 | Partial | Partially present, significant gaps |
| 2 | Adequate | Mostly present, minor gaps |
| 3 | Strong | Fully present, exemplary |
Evidence required
- Record inventory and retention periods
- A written account of which tests were unanswerable and why
Est. run time: 15 min
BCN-006 Trend — is the gap between reported performance and absence-fragility widening? consequence high
A single measurement establishes a level; the governance question is direction. A deployment whose reported performance and absence-fragility both rise quarter over quarter is accumulating dependence on correction while reporting improvement.
Pass criteria
- Both series are tracked over the same periods
- Divergence, if present, is reported to the body owning the deployment decision
- A threshold exists at which divergence triggers review
Fail indicators
- Only the performance series is tracked
- Divergence is visible in the data but has never been surfaced
- No threshold exists, so the trend can widen indefinitely without action
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Quarterly performance series
- Quarterly absence-degradation series
- Escalation threshold, if defined
Est. run time: 15 min
BCN-007 Discovery without a claim — does the operator find harm nobody complained about? consequence high
Standing and a working challenge route serve the party who challenges; they do nothing for the party who never challenges. A deployment whose only harm signal is an incoming claim is blind by construction to exactly the populations least able to file one. STD-06 §2.4 puts the duty to find silent harm on the deployer: differential error, reversal, and remedy rates, computed from records already held, on a cadence — not assembled after a complaint forces the question.
Pass criteria
- Error, reversal, and remedy rates are computed per affected population on a declared cadence
- A materially worse rate for any population is treated as a threshold breach, not as context
- The operator can name a harm its measurement found before any affected party reported it
Fail indicators
- Differential rates exist in retained records but are never computed or compared
- Harm is discovered only when an affected party challenges, and the operator cannot show otherwise
- Populations are merged in reporting so a worse rate for one is invisible inside the average
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Per-population rate series with the cadence declared
- The threshold that marks a materially worse rate and the action it triggers
- One example of a harm found by measurement rather than by claim
Est. run time: 20 min
BCN-008 Exhaustion-biased instrument — does the sensor run on the subject's last reserves? consequence critical
BCN-007 asks whether the operator finds harm nobody complained about. This case asks a prior question: whether the instrument would have found it if the reporting channel were the only sensor. A survey, an appeal, a complaint log, and a self-assessment all draw input from the subject's remaining attention, so the depleted answer least and the operator reads the gap as health. The bias runs one direction — more depletion produces a better-looking reading — and no better phrasing of the question repairs it. Law VIII asks whether an observation can change the governing state; this asks whether the observation is evidence at all.
Pass criteria
- Each instrument names who supplies its input and what that input costs the supplier
- Where the operator's own records answer the question, no person is asked
- Absence of reports is treated as a property of the instrument rather than a property of the population
- At least one burden signal is computed from the system's own trail, with no reporting step
- A score the operator produced is offered as evidence alongside an account, not as a reason to close it (STD-02 §5.4)
Fail indicators
- Complaint volume, appeal rate, or survey response is read as evidence that the population is unburdened
- Low response is attributed to satisfaction rather than to depletion
- The same defect would have gone undetected had the reporting channel been unavailable
- Detection depends on people spending discretionary energy the institution has already spent
- A passing burden ratio is cited as establishing that a person's account is unfounded
Scoring rubric (0-5)
| Score | Label | Description |
|---|---|---|
| 0 | Absent | Not present |
| 1 | Minimal | Mentioned but not operationalized |
| 2 | Basic | Exists but with significant gaps |
| 3 | Adequate | Meets minimum standard |
| 4 | Strong | Exceeds minimum, minor gaps |
| 5 | Exemplary | Production-grade, tested, auditable |
Evidence required
- Per-instrument statement of input source and supplier cost
- A burden signal computed from operational trail data with no reporting step
- The denominator used when report volume is compared across populations
Est. run time: 25 min