2.5 Metrics

Every agent runs a loop. It takes an input, reasons, acts, and folds the result back around. The chapter on loops said the metric is what the agent steers by — skip it and the agent searches forever, never knowing when to stop. That chapter spent its words on the acting half. This one develops the sensing half, because a loop with no honest sense of its own state cannot steer at all.

A metric senses. A playbook acts. A mechanism enforces. Those three make up a governed loop, and the metric comes first because the other two are blind without it. A playbook that fires on a number that measures the wrong thing does the wrong work. A mechanism that grades on a number too coarse to name the fault blocks the wrong commit. So the question that governs every metric in this chapter is not "can I measure this?" — almost anything can be counted — but "does this number drive a decision?"

2.5.1 Measure one level deeper

John Ousterhout gives the rule in one line ¹1. John Ousterhout, A Philosophy of Software Design (Yaknyam Press, 2018).: measure one level deeper than the number you think you want. A metric earns its place only when it drives a decision, and the surface number usually cannot; the deeper the measurement, the nearer it sits to the decision it must drive.

Think of the cloud bill. It tells you what the whole project spent last month. That number is real, and useless: it tells you to pay the invoice, and nothing about what to change — which service to slim, which call to batch, which cascade to cut. It affords the report; it cannot afford the engineering. Split the same bill by service and it starts to act. One service usually dominates, the rest trail off in a thin tail, and the dominant one names your first target. You measured one level deeper, and the deeper number carries a decision the shallow one buried.

Worked example — boot time, one level deeper

Say a service takes 8 seconds to boot. As one number it means only scale out — add instances to hide the wait. Measure one level deeper — 6 of those seconds are the GenAI client warming up, the rest a thin tail — and it means optimize the GenAI-client init first. Same latency, one level deeper, and now the number drives an engineering decision (fix the slow phase) instead of a reflexive scale-out. (Illustrative figures.)

The same trap hides in test coverage. "The suite covers 87% of lines" is the invoice-level number — a headline that reports and does not act. It cannot tell you which untested code matters, because a line in a throwaway string helper counts the same as a line in the job state machine. The rest of this chapter walks a spectrum of metrics from the hard, deterministic end to the soft, judgment-laden one, and the flagship in the middle is coverage measured one level deeper — coverage that names the gap instead of hiding it in a percentage. Table 2.5-1 collects the move, and the wrong turn each surface number invites.

The spectrum tracks the book's soft-versus-hard governance axis, and the pair splits cleanly. A hard metric is deterministic: a machine reads it, and the same input yields the same number every time. A soft metric is judgment-laden: it points a human or an agent at a place worth looking, but it cannot be reduced to a threshold that decides on its own. Both are worth having. The mistake is to treat one as the other — to block a build on a soft signal, or to leave a hard fact to a reviewer's memory. Figure 2.5-1 lays the metrics out from the hard, deterministic end to the soft.

2.5.2 The hard end: cold-start latency, machine-read and model-fed

Start at the deterministic end, where a metric is a millisecond number nobody argues about.

A serverless fleet scales to zero. When a request wakes a cold instance, the user waits while the container boots and the runtime loads. That wait is the cold-start latency, and the obvious way to read it, of course, is as the penalty of scaling from zero: measure it, and if it hurts, pay to keep a warm instance running.

The obvious reading was wrong, and measuring one level deeper is what showed it. The conformance service — a PDF validator wrapping veraPDF, a Java tool — carried a 4,057 ms warm floor, a wait a warm instance did nothing to remove. The deeper number named its own cause: veraPDF ran as a subprocess launched once per request, so every call paid a fresh JVM startup. The tax was the process launch itself, and no warm-instance lever could reach it. The fix was a re-architecture — keep the runtime resident instead of relaunching it — and the warm floor fell from 4,057 ms to 109 ms, a thirty-seven-fold cut that took nearly four seconds off every request. The .NET service told the same story smaller: 494 ms of per-request CLI launch, down to 103 ms once the runtime stayed resident. And the tempting surface explanation, that the container images were simply too big, was also wrong — a 433 MB service cold-started fine. The barrier was runtime initialization, not image size, and only the deeper measurement said so.

That is a metric read at the level where the decision lives: resident-runtime versus warm-instance is a different fork than the shallow cold-start number implied, and a fleet-average number would have buried it. But I found that tax by hand — spot-checking the one service whose latency looked wrong — and spot-checking does not scale. It catches the service that happens to draw your eye, misses the ones that do not, and says nothing about how the services interact. The better instrument is a model that holds every service's numbers and reasons over the paths between them, so the costly case falls out of the graph instead of out of a hunch. Because the sharpest question is the chain. A request that lands when every downstream service is cold triggers a cascade of cold-starts along the critical path. The revenue-core journey runs its steps in sequence; within a step, its calls fan out in parallel. So the worst case is a longest-path sum: each step contributes the slowest cold-start it waits on, and the path total is the sum of those per-step maxima. That sum, and neither the per-service number nor the fleet average, predicts the worst latency a real user can hit. It is the metric measured at the level where the decision lives.

Here the hard end shows its defining move: the number feeds the model. The cold-start measurements do not live in a dashboard a human reads and forgets. They land as typed fields on the deployment-topology model the fleet reasons through — a cold_start_ms and a warm_ms on each service's plane record. A recompute tool reads those fields and derives the longest-path worst case over the journey graph. When a later measurement writes a fresher number onto a service's annotation, the tool reads it live and the worst-case total refines on its own, no edit required.

Learn more about this governance mechanism: deployment-topology model.

The commit is automated. The tempting alternative is a runbook step — after a measurement run, remember to commit the new numbers. A reminder like that rots; someone forgets, and the model drifts from the fleet it describes. So the pipeline commits the refreshed measurements itself, under a bypass-prefixed housekeeping commit with no human in the loop, and the model's cold-start parameters stay current with zero manual action. Build the map's upkeep into the machine that changes the territory rather than asking a person to remember it — architecture over a reminder. The runbook note then documents an automated step instead of a chore.

Where this stands: the live-annotation path is built but not yet populated — today the tool runs off a snapshot table of measured values, each row citing the commit that measured it, because the fleet is mid-migration and no service carries the live annotation yet. The machinery to read the model live exists; the model is not fully fed. That is the ordinary shape of real work: the machinery lands first, and the data catches up. What matters for this chapter is the pattern — a hard metric, measured deterministically, written into a structured model, and consumed by a tool that turns it into a decision (which path, which service, which lever).

2.5.3 The flagship: coverage measured one level deeper

Now the middle of the spectrum, and the chapter's teaching example: the "measure one level deeper" rule applied to the number that most often gets it wrong.

Line coverage answers a syntactic question: did this line run under some test? That is a fact about source text, and it is genuinely useful. But it cannot answer the question an engineer actually has, which is semantic: is the claim this code makes exercised by a test? A file can sit at 95% line coverage while the one function that implements a critical invariant — call it INV-18 — is never called by any test at all. Its lines happen to be covered incidentally, swept up by an unrelated test that imports the module. The 95% is true. It is also a lie about the thing you care about, and the lie is invisible because a percentage has nowhere to put the gap.

The fix comes from joining two things the system already has. The first is a traceability graph that links each model claim — an invariant, a state-machine transition, a service-flow edge — to the code that implements it. The link points at an anchor: the entry point, the (path, symbol) of the function that carries the claim. The second is the ordinary coverage oracle, which knows exactly which source lines ran. Compose them:

anchor → its symbol → that symbol's line range → intersect with the covered-line set → is this model claim's code exercised?

That composition changes the unit of measurement. Line coverage counts lines. This counts model claims. The anchors supply the numerator's units; coverage supplies the ground truth of what ran. INV-18's anchor resolves to its function, and the function's lines are checked against what the tests actually executed. The metric then reports a named verdict rather than a bare percentage: INV-18's implementing code is not exercised by any test. An anonymous 87% becomes an actionable list of the exact model claims your suite walks past.

This is not a new invention, and it should not be. Run the genre check and the family has a name: requirements-based coverage — the discipline DO-178C mandates for avionics, and the shape the open-source Doorstop tool encodes with git-native requirement items linked to their tests. The system's traceability graph already adopted that schema. So there is no bespoke "model coverage" number to dream up. The work is to compute the requirements-based-coverage figure over a graph that was already built for it — and to add the one synthesis the off-the-shelf tools lack. Doorstop checks that a requirement has a linked test. It does not check that the linked test executes the requirement's code. Joining the trace edge to the live coverage oracle closes that last gap.

Learn more about this governance mechanism: requirements-based coverage.

The single primitive underneath — anchor_exercised, which resolves an anchor to a line range and intersects it with the covered set — supports several aggregations, each answering a different decision:

• Model-Element Coverage. Of the model claims that anchor real code, what fraction is exercised? This is the headline: it turns 95%-line-covered into a named list of unexercised invariants, each a fix target.
• Entry-Point Coverage. Of the code roots the models declare as "changes here mean drift," which are untested? This is the number to watch when refactoring — it tells you which anchored roots a regression would slip past.
• Chain Coverage. Does each model claim close end-to-end: code exists, a test names it as its verifier, and that test actually runs the code? This is the strictest cut, and it will report low at first, because few claims declare a verifying test today. That low number is the metric doing its job: it names the verification chains still to build.

The honesty layer matters as much as the aggregation. A single percentage lies when its denominator is fuzzy, so every figure reports a breakdown: exercised, not-exercised, unmeasurable (the anchor could not be resolved to code), and out-of-surface (a database-tier claim that is legitimately not code you can cover). "80%" then reads as what it is — 80% of the fifty measurable claims are exercised; twelve more are unmeasurable and owe a burn-down; eight are out of surface by design. That is a number that affords the engineering, because you can see which part of it you can act on.

The design is deliberately light. Computing all of this requires no change to any model — the anchors already carry their (path, symbol), and the covered-line set is a plain read of the coverage database the test runner already writes. The metric is a new consumer of two things the system already paid for: the traceability graph and the cross-language coverage oracle. Measuring one level deeper cost a join, not a rebuild.

2.5.4 The dual: draining unmodelled code

The flagship walks from a model claim to the code and asks whether a test exercises it. Turn the arrow around and a second metric appears. Walk from each test to the code it runs, and ask whether that code reaches any model-anchored symbol at all. A test whose exercised code reaches no model is an orphan — it guards behavior no model describes. The orphan rate, clustered by the unmodelled code the orphans touch, is a ranked list of the models still missing. Figure 2.5-2 shows what happened when the fleet worked that list down; Table 2.5-2 gives each re-run.

The metric is a model-discovery instrument, not a gate. Each re-run names the biggest remaining orphan cluster; an Epic then builds the missing structured model and its invariants for that subsystem; the tracer runs again and the orphan rate falls, because that subsystem's code now traces up to a model. Repeated cluster by cluster, the drain took the orphan rate from 56% at the pilot to 7.89% at the latest committed point — code-to-model coverage rising from 44% to about 92%. And once the residual dozen orphans are set aside as glue code — web-route and seam wiring with no model to miss — the genuine-orphan rate is 0%: every test whose code could trace to a model now does. The loop crossed its ≤10% target and, on genuine coverage, its tighter <5% goal. The metric drove the modeling, and the falling curve is the record of the modeling landing.

Two honesty notes bound the claim. The population is comparable across runs but not fixed: the denominator drifts from 144 to 152 tests as the suite evolves, and the first 56% point was scoped to the dispatch subsystem before the instrument widened. So read the curve as the same instrument re-run on a near-identical, evolving population, not as a fixed-N replay — and one interior point, the cost-rollup cluster's exact landing, is not separately enumerated in a committed record. A clean fixed-N series would re-run the tracer at each tagged commit over one frozen population; that measurement is pending and would sharpen the curve without changing its direction.

The same graph, read backward: a lens on accidental complexity

The link graph that drain walks has a second use, and it comes for free. Reading it forward — model claim to code — measured coverage. Read it backward and it starts to surface duplication the code hid. Two symbols that carry the same action but hang off different model nodes are a consolidation waiting to happen. A vocabulary mirrored across a service boundary — the same fact spelled twice, once on each side — is one model both sides should have read from instead. And a third pattern the loop turned up on its own: incomplete composition, where some paths reach the canonical seam and a sibling path quietly reaches a partial one. That last read caught a real resource leak on a cleanup path, not just a tidiness note.

So the model earns a second keep. The first is coverage — it tells you what is un-modeled. The second is a lens on accidental complexity — the traceability edges, inverted, point at duplication and half-finished composition the raw code does not advertise. This is an observation, not a tool. The loop fired it by hand, on some of the modeling efforts and not others, and left it a field note rather than a query surface — the yield has not yet earned the machinery, and building it before it does would be the over-engineering the method warns against. Named here because it is the kind of return a good model keeps giving after it has paid for itself: you drew the map to see what was un-governed, and it also shows you what was built twice.

2.5.5 The soft end: doc-derived tests

Slide to the far end of the spectrum, where the metric stops being a number a threshold can act on and becomes a judgment a human has to make.

A doc-derived test pins a behavior stated in a cited document or spec. The test carries a trailer naming the source it derived from, and when that source is edited, the test is regenerated so the pin and the prose stay in step. The metric it invites is: is this doc-claim pinned by a test? That question is genuinely soft. It has no clean denominator. "How many claims does this document make?" is a matter of reading — one reader finds six, another finds nine, and neither is wrong. You cannot reduce it to a percentage the way you can reduce line coverage, because the set of claims is not enumerable by machine. The signal points a reader at a document and says these behaviors deserve a pin; it does not decide.

Learn more about this governance mechanism: doc-derived test.

The softness runs deeper than a fuzzy denominator. A doc-derived test is only as good as the seam it tests against. If the code interleaves its decision logic with live I/O — a poll loop welded to a queue, a validator welded to a database read — there is no pure surface to assert a value against. The test then falls back to pinning the shape of the source: that a function exists, that a docstring reads a certain way. That kind of test stays green even when you gut the function's body. It measures nothing about behavior. So the doc-derived-test signal is inseparable from a judgment about readiness: are the types sound, is there a seam to test through, is the contract even documented? Only a human (or an agent reasoning as one) can answer that, which is what plants this metric firmly at the soft end. Treat it as hard — wire it to a gate that blocks on a coverage-of-doc-claims threshold — and you manufacture the very hollow tests it was meant to prevent.

A null result means nothing until the benchmark can discriminate

The most useful honesty in measurement is the one that catches you before you publish a comfortable non-result. An ablation was run to see whether giving the fleet a structured model helped it on a fixed set of tasks: run each task with the model and without it, and compare the scores. The comparison came back a null — no difference. The tempting reading is "the model did not help." The correct reading is that the instrument could not have shown help if help existed, because both arms scored at the ceiling. When the with-model arm and the without-model arm both nearly max the rubric, there is no room between them for an effect to appear. A null from an instrument with no discriminative headroom is not evidence against the hypothesis. It is not evidence for it either. It is simply uninformative, and treating it as a finding is the mistake the honesty is there to stop.

There is a structural reason the per-task cut was blind, and it generalizes past this one study. A task small enough to freeze into a benchmark cell is usually small enough to solve without the map. The value a structured model buys is a system-level one — the fix that lands once and holds everywhere, the drift a gate catches over months, a context-bounded fleet operating a codebase no context can hold at once — and none of that shows up inside a single self-contained gate-pass. So the right lesson is not "the model did not help." It is "measure the effect at the level where the effect lives," and be willing to say, in print, that a chosen instrument could not see it. The study that produced this null also retracted an earlier claim that had read the same null as a measured absence — correcting the record when the data would not support the sentence is the discipline that makes the rest of the numbers trustworthy.

2.5.6 The model runs the machinery

The hard end and the flagship follow the same pattern.

The MBSE models do more than stay in sync with the code as documentation: they are consumed at runtime to produce the metrics. The cold-start numbers land as typed fields on the deployment-topology model — no spreadsheet in sight — and a tool reads the model to derive the longest-path worst case. The coverage flagship walks the traceability graph, follows each model claim to its anchor, and measures that, rather than scanning source text for functions that look important. In both, the model is not a picture of the machinery. The model is the machinery — the thing the metric tool reads to know what to measure and how to weigh it.

The sharpest instance ties metrics back to mechanisms. A pre-commit gate has to decide whether a lint finding should block a commit. The naive gate blocks on any finding anywhere, which drowns an agent in pre-existing debt it did not cause. The better gate grades each finding by its distance from the change in the typed component graph, and the grade decides the gate's behavior:

Learn more about this governance mechanism: model-graded finding severity.

• HARD — the finding sits on a file this commit touched, or is directly caused by a touched input. Block. This is the commit's own debt, and it fails closed.
• SOFT — the finding sits in the same component as a touched file, but not on a touched file. Report it, name it, ask the agent whether the change plausibly caused it — but do not block. Blocking here would make concurrent agents race to fix the same pre-existing problem and collide.
• SILENT — the finding sits in a different component entirely. Suppress it at commit time; the whole-tree deploy gate remains the backstop.

The grade is a metric — a distance, read from the model at check time — and it drives a sensor's decision directly. Same finding, three different gate behaviors, chosen by consulting the component graph about how far the finding is from the change. Here the model earns its keep at the moment a commit is made: it is queried, live, to grade a gate — the machinery, run by the model.

2.5.7 A compact reference

Table 2.5-3 collects every metric — the decision each drives, where it sits on the hard-soft axis, and who reads it.

Read the table top to bottom and the spectrum is visible in one glance: the top rows are stopwatch-hard and machine-consumed, the middle rows turn a hard measurement onto a semantic unit, and the bottom rows aim a reader who still has to judge. Every row names a decision, because a metric that names no decision was measured too shallow — and a loop steering by a shallow number is a loop that cannot steer.

This table steers a single loop iteration. A mature Governed Engineering Environment is operated one level up too, through a small set of formative and summative metrics — a few you steer by while the work is in flight, a few you certify the result with at maturity, a few that serve both. The complete operator's reference, every metric with what it counts and its healthy direction, is collected in Appendix D.