Skip to content

B.4 Measuring MAGE

Read outcomes before diagnostics. The operating question is whether delegated work produces durable, trustworthy progress at a cost the environment can sustain. The five primary readings in Table B.4-1 keep that question visible. They are not a composite score: a gain in throughput does not cancel an unacceptable escape, and a large stock of models does not compensate for models that cannot answer their claimed questions.

Table B.4-1. Read outcomes and the condition of the environment together; use diagnostic counts to investigate changes in these readings.
ReadingQuestionRecord
Durable throughputHow much accepted work survives without reopening, rollback, or repeated repair?Accepted units, observation window, and later repair.
Defect escapeWhat incorrect or policy-violating work crosses the boundary intended to catch it?The obligation, boundary, affected revision, and discovery.
Human-attention burdenWhat attention does each durable unit require?Reconstruction, review, exceptions, recovery, and deliberate judgment.
Representation healthCan relied-upon models still answer their claimed questions?Correspondence, relevant coverage, and unresolved drift.
Engineering-capital returnDoes inherited structure repay its construction and carrying cost?Later uses, avoided work, maintenance, and retirement candidates.

Before changing the environment, record several comparable work units under the existing process. Preserve their acceptance criteria, completion evidence, human interventions, and subsequent repairs. Use the same unit and observation window after introducing a model or control. Where work differs materially in complexity or consequence, report those differences rather than attributing every change in throughput or attention to MAGE. A before-and-after comparison can suggest whether the investment helped; it does not by itself establish causation.

Agree on the unit, boundary, and observation window before interpreting a trend. A landed change, completed work order, and released capability are different units. Keep their denominators visible and distinguish changes in the workload from changes in the machinery. Retain the revisions and evidence behind consequential observations. A dashboard can help locate a change in behavior; it cannot supply a comparison that the underlying records do not support.

Define the Five Readings

Durable throughput counts accepted work that remains useful after the initial completion claim. Choose a window appropriate to the work and follow reopenings, rollbacks, and repeated repairs within it. Report recent work whose durability is not yet observable separately from work that has survived the window. A later change in requirements is not necessarily evidence that the original work failed; repeated repair of the original obligation is. Commit count and immediate ticket closure omit this distinction. The desired result is more accepted capability without a proportional increase in intervention.

Defect escape concerns a particular obligation and boundary. Record what crossed, which mechanism was supposed to prevent it, and how it was eventually discovered. A failed test before admission demonstrates detection; the same defect found after the relevant boundary demonstrates an escape. Compare rates only when the governed obligations and opportunities for observation are sufficiently comparable. More reported defects may reflect better observation rather than worse production, and no reported defects may reflect weak observation. State the limits of the evidence instead of converting absence of reports into confidence.

Human-attention burden includes the work hidden behind an apparently autonomous result. Record repeated reconstruction, routine inspection, exception handling, recovery, and consequential judgment, using time where it is practical and intervention records where it is not. Relate the burden to durable output rather than generated output. Keep intentional engineering decisions visible: choosing new obligations or interpreting ambiguous evidence is different from explaining the same repository convention again. A reduction is valuable when the work has been made unnecessary or carried reliably by the environment, not merely when it stops being recorded.

Representation health depends on the claim each representation makes. A descriptive model must correspond to the system it describes; a normative model states intent the system must satisfy; a derived view depends on its source and transformation. Historical work records have the different lifetime described earlier. Examine drift findings, failed queries, missing consequential coverage, and the evidence used to reconcile them. Freshness means correspondence after relevant change, not simply a recent edit date. A synchronized filename and a valid link do not settle a disagreement about meaning.

Engineering-capital return asks what later work inherits and what that inheritance costs. Retain concrete examples: a dependency no longer reconstructed, a recurring failure prevented, a larger task made tractable, or knowledge available after its original author has left. Account for construction, execution, understanding, reconciliation, and maintenance. A model may repay its cost without preventing a defect, just as a control may justify itself through a consequential risk rather than frequent use. Reuse counts alone cannot establish either return. Review structures whose carrying cost has outlived their purpose, and retire them when appropriate.

Use Diagnostics to Explain, Not to Score

Churn, model coverage, support ratio, control count, closure rate, and queue length can help explain a primary reading. Churn can reveal repeated repair or useful restructuring. Coverage can expose a consequential gap or merely code below the model’s grain. A support ratio describes the amount of surrounding engineering machinery relative to implementation; it does not establish that the machinery pays. More controls can indicate greater protection or accumulated interference. Inspect the underlying work before treating any direction as healthy. The empirical observations in Chapter 5 are evidence from particular settings, not operating targets.

Include the health of observation in the review. Did scheduled checks run? Which revisions did they inspect? Could a check fail without anyone seeing or acting on its result? Were exceptions closed through verified repair or only acknowledgment? An empty findings list is uninterpretable when the check never ran. Similarly, an uncalibrated visual indicator is status, not a quantitative measurement. Preserve enough of the observation process to distinguish a quiet system from a silent sensor.

Review the readings at a cadence suited to the work and after a consequential incident. Investigate the explanations before prescribing another model or gate. A rise in attention may be temporary investment in new capability; declining throughput may be the deliberate cost of exposing previously hidden defects. Decide whether to repair an existing structure, extend coverage, retire unnecessary machinery, or keep the current arrangement. Measurement earns its place through those decisions and through the evidence that later shows whether they helped.

© James C. Davis, 2026–present