Delivery policies, release risk, and the Four Keys — an interactive experiment.
Teach with MAGE ECE 30861 · Act III Compute Environments & Deployment
Engineering problem
How do testing and release policies trade delivery speed against defect exposure?
A team lands changes all day. Every policy between a commit and a user buys one thing with
another: deeper testing buys evidence with time, a blocking gate buys correctness with
throughput, a staged rollout buys a smaller blast radius with delay. This laboratory lets you
change one policy at a time and measure what it cost and what it bought.
Illustrative simulation; not calibrated to an industrial organisation.
Background
A release pipeline is a sequence of stages, and every stage is a queue. A
change is built and tested, admitted or rejected, packaged into a release, and finally served
to users. Each stage has a finite capacity — a number of parallel test lanes, a number of
release slots. Arrivals are irregular and service takes time, so work waits, and a stage
close to its capacity grows a waiting line faster than the load that feeds it.
Testing and enforcement are different mechanisms. A test suite produces a
signal: this change looks broken. An admission policy decides what that signal is
allowed to do. Advisory checks warn and the change ships anyway; required checks let the same
signal stop it. The suite is identical in both cases — what differs is the authority attached
to its verdict. A pipeline can test well and enforce nothing, and it will ship the defects its
own tests found.
Deeper testing lengthens service time, and service time compounds into queueing
delay. A thorough suite runs longer than a smoke suite, so every change occupies a
lane for longer. While the stage stays under capacity, lead time rises by roughly the extra
minutes. Once the longer suite pushes the stage past capacity, the line never drains
and lead time rises without bound. The same policy change has two qualitatively different
outcomes depending on which side of capacity it lands on.
Staged release changes who meets a defect, not whether one exists. Deploying
to the whole fleet exposes every user to whatever the release carries. A canary serves a small
share of traffic first and holds there for a soak period, so a defect that manifests often is
likely to be noticed while only that share is exposed. The canary does not make the defect
less likely. It bounds the requests served before somebody notices — and only when the defect
manifests often enough to appear in the soak's traffic at all.
Delivery performance and software correctness are separate measurements. How
often a team deploys, how long a change takes to reach production, how often a deployment
fails and how fast service is restored all describe the delivery process. None of
them says whether the software is correct or the change was worth making. A pipeline can ship
ten times a day, restore in minutes, and expose users continuously — because a defect nobody
detected appears in no failure rate.
The model
Figure 1. The pipeline the simulator runs. The four policies you
can change are marked above the stages they govern. Monitoring and recovery sit outside the
delivery line because they happen after a change is delivered, and are measured
separately.
Model assumptions
This instrument measures delivery performance and release risk. Its
four delivery measures are the Four Keys — deployment frequency,
lead time for changes, change failure rate, time to restore service — the four the
course reading names, and not an account of current DORA measurement practice, which has
moved on.
They describe selected outcomes of the delivery process. They do not establish
whether the organisation is producing valuable or correct software.
Changing a policy resets the factory, and the seed does not change. The
same team commits the same changes carrying the same defects, so switching one policy is a
controlled comparison rather than a fresh draw.
Thorough testing never catches everything. The deepest suite's coverage
factor is 0.92, not one, so some defects always reach production.
Table 1 · Model parameters
Table 1: the four policies you can change
Policy
Values
What changing it does
Change arrival rate
Low 4 · Medium 14 · High 44 changes/day
Sets the demand on the pipeline.
Test depth
Smoke 8 · Standard 42 · Thorough 155 min
Buys evidence with service time at the build and test stage.
Admission policy
Advisory · Required
Decides whether a failing check may stop a change.
Release strategy
Immediate · Canary · Blue-green
Decides how much traffic meets a release before anyone has looked at it.
Deployment batch size is fixed at one change per release. Everything else the model needs
— lanes, defect rate, traffic, monitoring quality, soak and rollback durations —
is held at the values printed in Advanced controls.
Table 2 · Observed quantities
Table 2: what the simulation reports and what each quantity counts
Quantity
What it counts
Reported as
Lead time for changes
First commit to first arrival in production, including any rework.
Principal
Defects reaching production
Defects that served real traffic, whether or not anybody noticed.
Principal
User exposure
Requests served by a release while it carried a live defect.
Principal
Deployment frequency
Releases that reached real users, per measured day.
Secondary
Change failure rate
Failed deployments divided by deployments — not by changes.
Secondary
Time to restore service
Detection to rollback complete. The clock starts when somebody noticed.
Secondary
The experiment
Procedure
1 · Predict
Read an experiment below and write your prediction down before you touch anything. A
prediction you did not commit to is not a prediction.
2 · Run
Press Set baseline, then Run 90 days, and record the three
principal results. Press the experiment's other button and run 90 days again.
3 · Explain
Compare the two records, then answer the experiment's engineering question. The seed does
not change between runs, so the same team committed the same defects in both.
Experiment 1 Testing and delivery
Baseline
High arrival · Standard test depth · Required admission ·
Immediate release
Changing a policy resets the factory — these are experiments watched from
an empty pipeline, not perturbations of a running one. The seed survives the reset, so the same
changes arrive at the same times carrying the same defects. Speed changes how fast you watch,
never what happens: 1× is one simulated hour per second.
Principal results
Lead time for changes—
Defects reaching production—
User exposure — requests served—Median per failed deployment:
—
Secondary measurements
Deployment frequency—
Change failure rate—
Time to restore service—
The failure rate is a measurement, not the truth. A deployment carrying a defect
nobody detected is not in its numerator. That is why defects reaching production is a
principal result and the failure rate is not.
The factory, running
Simulated time
—
Build + test stage
—
Release channel
—
model Both utilisation figures are offered load divided by nominal
capacity, computed from the policy tables rather than measured from the run. They ignore the
work that rework and rollback generate, so a factory can sit below one on both and still grow a
backlog.
Work in the factory, over time
(secondary visualisation)
Three series, each labelled on the chart and drawn with its own line style, so the picture reads
without colour. A queue that climbs and never comes back down is a stage past its capacity.
Interpretation
Name the policy that moved delivery speed most, and the policy that moved
defect exposure most. They are not the same policy.
Find a configuration that improves one of the secondary measurements while making the
factory worse for its users, and say which measurement lied.
In Experiment 2, the test suite signalled the same defects under both policies. State in one
sentence what the required gate bought and what it charged for it.
In Experiment 3, the change failure rate barely moved while user exposure fell sharply.
Explain why both of those are the correct behaviour of a canary.
You are asked to recommend one policy change to a team that is shipping too slowly. Which
measurement would you ask to see first, and what would you refuse to conclude from it?
Further investigation
The defect ledger — what becomes of a defect
Figure 2. The stochastic model behind the ledger. Testing either
catches a defect before any user meets it, or it does not — and after that, detection and
restoration are separate events with their own durations.
A defect is tracked as five events, never collapsed into one: introduced,
detected before release, exposed in production,
detected after release, service restored. Two rows sit beside
them because collapsing those would hide a policy: the tests’ signal is recorded
separately from the gate’s block (so an advisory policy is visible as a signal
that shipped anyway), and a defect removed by someone else’s rollback is counted as
removed, not restored — nobody ever knew it was there.
Capacity — hold the pipeline fixed and raise the arrival rate
The same policies, the same two build–test lanes and one release slot, run independently at
each arrival rate on the chart. Every point is a full seeded simulation.
The sweep holds everything except the arrival rate, including the seed offsets, so the curve is
a function of load alone. A pipeline has two capacities — the build–test
lanes and the release channel — and whichever one the policies make scarcer is the one that
bends first.
Two things to read carefully. The curve usually bends well to the left of the dashed
capacity lines, because those lines count only the first pass through each
stage: a rollback sends its whole batch back, so a factory that fails often spends real capacity
recovering from itself. And beyond the knee the measured lead time flattens
— an artefact, not a recovery. A queue past its capacity has no steady state, so the
average wait is bounded by how long the simulation ran. In an unstable system, the number
you measure depends on how long you watch.
Explore further — the full configuration library
Beyond the three guided experiments. Each card sets a whole configuration and resets the
factory; some of them move settings the four primary controls hold fixed, which you can see in
Advanced controls after loading one.
The canary depends on the defect, not on the strategy
Detection in production is a Poisson process whose rate is
affected requests/hour × distinguishability × monitoring
quality, so the chance of catching a defect inside a soak of length
S is 1 − e−rate×S.
A canary enters only through the exposed fraction. Run these two back to back: the first makes a
canary look like a triumph, the second like a tax, and both are the same code.
Advanced controls
Everything the four primary controls hold fixed. Changing any of these resets the factory, and
takes the exercise outside the three guided experiments.
Deployment batch size
changes per release — fixed at 1 for the guided experiments
Warm-up excludes the first days from every aggregate. It is zero by default and
that is deliberate: a factory past its capacity has no steady state to warm up into,
and hiding the transient would hide the finding.
Model reference — equations, limitations, and the engine
What the model is
A seeded discrete-event simulation of one delivery pipeline. Changes arrive as a
Poisson process; each is tested in one of a fixed number of lanes; an admission policy decides
whether a failing change proceeds; admitted changes are batched into releases; a release strategy
decides how the batch meets users; production monitoring may notice a defect, after which the
release is rolled back and its changes return through the pipeline.
Every number on this page comes from that one engine. Nothing is a lookup table, and no outcome
is predetermined — which is why two policies can swap places when you change the defect
population or the traffic volume.
Limitations — what this simulation does not establish
No figure here measures a real organisation. Every probability and duration
is a teaching-scale assumption, printed on this page, not an industrial measurement.
A single run is not an expected value. The model is stochastic. A common
seed makes two configurations comparable — the same changes arrive carrying the same
defects — but it does not remove sampling variability. Two seeds will not give the same
numbers, and a difference smaller than the spread across seeds establishes nothing. Change the
seed in Advanced controls and re-run before trusting a narrow gap.
Feature flags are not modelled. They decouple deploy from release, which
would need a separate exposure control.
A rollback reverts the whole batch. There is no partial rollback.
A later release does not fix an earlier defect. An undetected defect keeps
accruing exposure until somebody finds it. That is deliberate, and it is where most of the
user impact comes from.
Dependencies between changes, review time, and whether a change was worth
making are all outside the model.
The five events, and why they are kept apart
A defect is an object with its own lifecycle. The interface reports three summary numbers;
the engine keeps every one of these distinct, because collapsing any two would make a policy
question unanswerable:
Event
Fires when
What it lets you ask
1 · introduced
a defective change is committed
how many defects does this team produce?
2 · detected before release
a pre-production control found it and the change did not ship
what did the gate actually keep out?
3 · exposed in production
the change first served real traffic
what reached users, whatever the tests said?
4 · detected after release
monitoring or a user report noticed it
what did we find, and how fast?
5 · service restored
the rollback finished
how long did users carry it?
signalled by tests
the suite failed on this defect
the signal, separate from the block — which is how an advisory
policy becomes measurable
removed from production
a rollback triggered by a different defect took it away
not a restoration: nobody knew it was there
Events 2 and 3 are mutually exclusive. Event 4 implies event 3. Event 5 implies event 4.
A defect can be exposed and never detected, in which case it is never restored
and never appears in the change failure rate. The test suite checks every one of these on every
release strategy.
How a defect behaves
Each defect draws three attributes from a four-class mixture at the moment it is committed,
independently of every policy. The classes are teaching-scale assumptions, not measurements of
any real population, and they are printed here so you can see what the simulator believes:
Class
Share
Test detectability
Manifests on
Distinguishable
Character
Test detectability is the probability a fully thorough suite signals
it. The configured depth multiplies that by a coverage factor — and the deepest suite's
factor is 0.92, not one, so thorough testing never eliminates
all defects. There is no branch anywhere that sets the escape probability to zero.
Detection in production
Detection is a Poisson process. Its rate is built from affected traffic, not from the release
strategy:
So the chance of catching a defect inside a canary soak of length
S is 1 − e−rate×S.
A canary enters only through the exposed fraction. That is the whole of its
effect, and it is why the answer depends on the defect: at twenty thousand requests an hour a
defect manifesting on 30% of them is caught in the soak essentially always, and one manifesting
on 1 request in 100,000 essentially never. Nothing hard-codes that a canary improves
the change failure rate.
The bug-report channel is proportional to affected users rather than a flat background
rate, because a user can only report what they experienced. A flat rate would quietly guarantee
that every defect is eventually found, which would delete the “exposed but never
detected” case the model exists to show.
Blue-green verification uses the same defect attributes against a fixed number of
synthetic requests, so it misses a rare defect for the same reason a canary does — not
enough requests exercise it. What it does differently is expose nobody while it looks.
How the measures are computed
Measure
Definition used here
Lead time for changes
first commit to first arrival in production, including any rework. One
observation per change: a rolled-back change shipped again does not produce a second, or
the mean would depend on how often the factory fails.
Defects reaching production
defects whose change served real traffic, counted whether or not anybody ever noticed.
Split in the readout into those detected after release and those still live and
undetected.
User exposure
requests served by a defective release while it is live, integrated over the exposed
traffic share. Reported as a total, a per-day rate, and the median per failed
deployment — the last because the total is dominated by the long tail of
slowly-detected defects, so a canary moves the median far more than the mean.
Deployment frequency
releases that reached real users, per measured day. A blue-green release aborted at
verification never reached users, so it is not a deployment.
Change failure rate
failed deployments ÷ deployments — not changes. A deployment
fails if a defect it carries is detected in production.
Time to restore service
detection to rollback complete. The clock starts when somebody noticed, not when
the defect was exposed.
Fixed parameters and reproducibility
The engine is a pure module published as window.ReleaseFactory with no
reference to the page. It uses six independent seeded generators — arrivals, defect
attributes, rework regressions, test verdicts, durations, production detection — and no
unseeded random source and no clock, so the same seed and parameters replay the same event trace
exactly, and switching one policy leaves the others’ draws where they were. Its test suite
extracts this element from the published file, so the tested code and the code you are running
are the same bytes. Every value the four primary controls hold fixed is listed, with its
default, in Advanced controls.