The Release Factory

Delivery policies, release risk, and the Four Keys — an interactive experiment.

Teach with MAGE
ECE 30861 · Act III
Compute Environments & Deployment

Engineering problem How do testing and release policies trade delivery speed against defect exposure?

A team lands changes all day. Every policy between a commit and a user buys one thing with another: deeper testing buys evidence with time, a blocking gate buys correctness with throughput, a staged rollout buys a smaller blast radius with delay. This laboratory lets you change one policy at a time and measure what it cost and what it bought.

Illustrative simulation; not calibrated to an industrial organisation.

Background

A release pipeline is a sequence of stages, and every stage is a queue. A change is built and tested, admitted or rejected, packaged into a release, and finally served to users. Each stage has a finite capacity — a number of parallel test lanes, a number of release slots. Arrivals are irregular and service takes time, so work waits, and a stage close to its capacity grows a waiting line faster than the load that feeds it.

Testing and enforcement are different mechanisms. A test suite produces a signal: this change looks broken. An admission policy decides what that signal is allowed to do. Advisory checks warn and the change ships anyway; required checks let the same signal stop it. The suite is identical in both cases — what differs is the authority attached to its verdict. A pipeline can test well and enforce nothing, and it will ship the defects its own tests found.

Deeper testing lengthens service time, and service time compounds into queueing delay. A thorough suite runs longer than a smoke suite, so every change occupies a lane for longer. While the stage stays under capacity, lead time rises by roughly the extra minutes. Once the longer suite pushes the stage past capacity, the line never drains and lead time rises without bound. The same policy change has two qualitatively different outcomes depending on which side of capacity it lands on.

Staged release changes who meets a defect, not whether one exists. Deploying to the whole fleet exposes every user to whatever the release carries. A canary serves a small share of traffic first and holds there for a soak period, so a defect that manifests often is likely to be noticed while only that share is exposed. The canary does not make the defect less likely. It bounds the requests served before somebody notices — and only when the defect manifests often enough to appear in the soak's traffic at all.

Delivery performance and software correctness are separate measurements. How often a team deploys, how long a change takes to reach production, how often a deployment fails and how fast service is restored all describe the delivery process. None of them says whether the software is correct or the change was worth making. A pipeline can ship ten times a day, restore in minutes, and expose users continuously — because a defect nobody detected appears in no failure rate.

The model

Figure 1: the modelled release pipeline, from a code change to production A pipeline runs left to right in five stages: code changes, build and test, an admission gate, deploy, and production, where users are exposed. Each of the first four stages is annotated with the policy that governs it — arrival rate, test depth, admission policy and release strategy. A dashed path returns from the admission gate to build and test, labelled blocked then reworked. Below production, an arrow leads to a separate stage labelled monitor then restore, annotated detected after release, drawn outside the main line because it happens after delivery rather than as part of it. A bracket spans code changes to production and is labelled lead time, commit to production. arrival rate test depth admission policy release strategy Code changes Build + test Admission gate Deploy Production users exposed blocked, then reworked Monitor, then restore detected after release lead time · commit to production
Figure 1. The pipeline the simulator runs. The four policies you can change are marked above the stages they govern. Monitoring and recovery sit outside the delivery line because they happen after a change is delivered, and are measured separately.

Model assumptions

  • This instrument measures delivery performance and release risk. Its four delivery measures are the Four Keys — deployment frequency, lead time for changes, change failure rate, time to restore service — the four the course reading names, and not an account of current DORA measurement practice, which has moved on.
  • They describe selected outcomes of the delivery process. They do not establish whether the organisation is producing valuable or correct software.
  • Changing a policy resets the factory, and the seed does not change. The same team commits the same changes carrying the same defects, so switching one policy is a controlled comparison rather than a fresh draw.
  • Thorough testing never catches everything. The deepest suite's coverage factor is 0.92, not one, so some defects always reach production.

Table 1 · Model parameters

Table 1: the four policies you can change
PolicyValues What changing it does
Change arrival rate Low 4 · Medium 14 · High 44 changes/day Sets the demand on the pipeline.
Test depth Smoke 8 · Standard 42 · Thorough 155 min Buys evidence with service time at the build and test stage.
Admission policy Advisory · Required Decides whether a failing check may stop a change.
Release strategy Immediate · Canary · Blue-green Decides how much traffic meets a release before anyone has looked at it.

Deployment batch size is fixed at one change per release. Everything else the model needs — lanes, defect rate, traffic, monitoring quality, soak and rollback durations — is held at the values printed in Advanced controls.

Table 2 · Observed quantities

Table 2: what the simulation reports and what each quantity counts
QuantityWhat it counts Reported as
Lead time for changes First commit to first arrival in production, including any rework. Principal
Defects reaching production Defects that served real traffic, whether or not anybody noticed. Principal
User exposure Requests served by a release while it carried a live defect. Principal
Deployment frequency Releases that reached real users, per measured day. Secondary
Change failure rate Failed deployments divided by deployments — not by changes. Secondary
Time to restore service Detection to rollback complete. The clock starts when somebody noticed. Secondary

The experiment

Procedure

  1. 1 · Predict Read an experiment below and write your prediction down before you touch anything. A prediction you did not commit to is not a prediction.
  2. 2 · Run Press Set baseline, then Run 90 days, and record the three principal results. Press the experiment's other button and run 90 days again.
  3. 3 · Explain Compare the two records, then answer the experiment's engineering question. The seed does not change between runs, so the same team committed the same defects in both.

Experiment 1 Testing and delivery

Baseline
High arrival · Standard test depth · Required admission · Immediate release
Change
Test depth: Standard → Thorough
Held constant
Arrival rate, admission policy, release strategy, batch size, seed
Predict
What happens to lead time? To the number of defects reaching production? Which moves further?
Compare
Lead time · defects reaching production · user exposure
The question it answers
Under what conditions is the additional testing time justified?

Experiment 2 Enforcement

Baseline
High arrival · Standard test depth · Required admission · Immediate release
Change
Admission policy: Required → Advisory
Held constant
Arrival rate, test depth, release strategy, batch size, seed
Predict
The same suite runs and signals the same defects in both configurations. Predict what changes anyway, and what does not.
Compare
Defects reaching production · user exposure · lead time
The question it answers
What does an enforcing gate add that a test suite alone cannot?

Experiment 3 Exposure

Baseline
High arrival · Standard test depth · Required admission · Immediate release
Change
Release strategy: Immediate → Canary
Held constant
Arrival rate, test depth, admission policy, batch size, seed
Predict
Does the count of defects reaching production fall? Does user exposure fall? What happens to lead time?
Compare
User exposure · defects reaching production · lead time
The question it answers
When is a staged release worth the lead time it costs?

Apparatus — the four policies

Change arrival rate — how much work arrives
Test depth — how long a change holds a lane
Admission policy — may a failing check stop a change?
Release strategy — how much traffic meets a release
Speed
Download HTML

Changing a policy resets the factory — these are experiments watched from an empty pipeline, not perturbations of a running one. The seed survives the reset, so the same changes arrive at the same times carrying the same defects. Speed changes how fast you watch, never what happens: 1× is one simulated hour per second.

Principal results

Lead time for changes —  
Defects reaching production —  
User exposure — requests served —   Median per failed deployment: —

Secondary measurements

Deployment frequency —  
Change failure rate —  
Time to restore service —  

The failure rate is a measurement, not the truth. A deployment carrying a defect nobody detected is not in its numerator. That is why defects reaching production is a principal result and the failure rate is not.

The factory, running

Simulated time
—
Build + test stage
—
Release channel
—

model Both utilisation figures are offered load divided by nominal capacity, computed from the policy tables rather than measured from the run. They ignore the work that rework and rollback generate, so a factory can sit below one on both and still grow a backlog.

Work in the factory, over time (secondary visualisation)

Three series, each labelled on the chart and drawn with its own line style, so the picture reads without colour. A queue that climbs and never comes back down is a stage past its capacity.

Interpretation

  1. Name the policy that moved delivery speed most, and the policy that moved defect exposure most. They are not the same policy.
  2. Find a configuration that improves one of the secondary measurements while making the factory worse for its users, and say which measurement lied.
  3. In Experiment 2, the test suite signalled the same defects under both policies. State in one sentence what the required gate bought and what it charged for it.
  4. In Experiment 3, the change failure rate barely moved while user exposure fell sharply. Explain why both of those are the correct behaviour of a canary.
  5. You are asked to recommend one policy change to a team that is shipping too slowly. Which measurement would you ask to see first, and what would you refuse to conclude from it?

Further investigation

The defect ledger — what becomes of a defect

Figure 2: what becomes of a change that carries a defect A tree runs left to right. A change contains a defect and enters testing. Testing has two outcomes. On the upper branch the defect is detected, the change is blocked and reworked, and no user is exposed. On the lower branch the defect is missed; the change is deployed and users are exposed, after which the defect is either detected and the service restored, or never detected at all, which is drawn as a separate leaf. The missed branch and everything downstream of it are drawn in the failure colour and also labelled in words. A note at the foot reads that the branch probability is the test depth's coverage factor, an assumption printed on this page and not a measured industrial rate, and that the five events on the ledger beside this figure are the five points on this tree. Change contains a defect Testing detected Blocked, reworked no user exposed missed Deployed users exposed Detected, then restored time to restore starts here Never detected The branch probability is the test depth’s coverage factor: an assumption printed on this page, not a measured industrial rate. The ledger’s five events are the five points on this tree.
Figure 2. The stochastic model behind the ledger. Testing either catches a defect before any user meets it, or it does not — and after that, detection and restoration are separate events with their own durations.

A defect is tracked as five events, never collapsed into one: introduced, detected before release, exposed in production, detected after release, service restored. Two rows sit beside them because collapsing those would hide a policy: the tests’ signal is recorded separately from the gate’s block (so an advisory policy is visible as a signal that shipped anyway), and a defect removed by someone else’s rollback is counted as removed, not restored — nobody ever knew it was there.

Capacity — hold the pipeline fixed and raise the arrival rate

The same policies, the same two build–test lanes and one release slot, run independently at each arrival rate on the chart. Every point is a full seeded simulation.

The sweep holds everything except the arrival rate, including the seed offsets, so the curve is a function of load alone. A pipeline has two capacities — the build–test lanes and the release channel — and whichever one the policies make scarcer is the one that bends first.

Two things to read carefully. The curve usually bends well to the left of the dashed capacity lines, because those lines count only the first pass through each stage: a rollback sends its whole batch back, so a factory that fails often spends real capacity recovering from itself. And beyond the knee the measured lead time flattens — an artefact, not a recovery. A queue past its capacity has no steady state, so the average wait is bounded by how long the simulation ran. In an unstable system, the number you measure depends on how long you watch.

Explore further — the full configuration library

Beyond the three guided experiments. Each card sets a whole configuration and resets the factory; some of them move settings the four primary controls hold fixed, which you can see in Advanced controls after loading one.

The canary depends on the defect, not on the strategy

Detection in production is a Poisson process whose rate is affected requests/hour × distinguishability × monitoring quality, so the chance of catching a defect inside a soak of length S is 1 − e−rate×S. A canary enters only through the exposed fraction. Run these two back to back: the first makes a canary look like a triumph, the second like a tax, and both are the same code.

Advanced controls

Everything the four primary controls hold fixed. Changing any of these resets the factory, and takes the exercise outside the three guided experiments.

Deployment batch size changes per release — fixed at 1 for the guided experiments

Warm-up excludes the first days from every aggregate. It is zero by default and that is deliberate: a factory past its capacity has no steady state to warm up into, and hiding the transient would hide the finding.

Model reference — equations, limitations, and the engine

What the model is

A seeded discrete-event simulation of one delivery pipeline. Changes arrive as a Poisson process; each is tested in one of a fixed number of lanes; an admission policy decides whether a failing change proceeds; admitted changes are batched into releases; a release strategy decides how the batch meets users; production monitoring may notice a defect, after which the release is rolled back and its changes return through the pipeline.

Every number on this page comes from that one engine. Nothing is a lookup table, and no outcome is predetermined — which is why two policies can swap places when you change the defect population or the traffic volume.

Limitations — what this simulation does not establish

The five events, and why they are kept apart

A defect is an object with its own lifecycle. The interface reports three summary numbers; the engine keeps every one of these distinct, because collapsing any two would make a policy question unanswerable:

EventFires whenWhat it lets you ask
1 · introduceda defective change is committed how many defects does this team produce?
2 · detected before release a pre-production control found it and the change did not ship what did the gate actually keep out?
3 · exposed in productionthe change first served real traffic what reached users, whatever the tests said?
4 · detected after releasemonitoring or a user report noticed it what did we find, and how fast?
5 · service restoredthe rollback finished how long did users carry it?
signalled by teststhe suite failed on this defect the signal, separate from the block — which is how an advisory policy becomes measurable
removed from production a rollback triggered by a different defect took it away not a restoration: nobody knew it was there

Events 2 and 3 are mutually exclusive. Event 4 implies event 3. Event 5 implies event 4. A defect can be exposed and never detected, in which case it is never restored and never appears in the change failure rate. The test suite checks every one of these on every release strategy.

How a defect behaves

Each defect draws three attributes from a four-class mixture at the moment it is committed, independently of every policy. The classes are teaching-scale assumptions, not measurements of any real population, and they are printed here so you can see what the simulator believes:

ClassShareTest detectability Manifests onDistinguishableCharacter

Test detectability is the probability a fully thorough suite signals it. The configured depth multiplies that by a coverage factor — and the deepest suite's factor is 0.92, not one, so thorough testing never eliminates all defects. There is no branch anywhere that sets the escape probability to zero.

Detection in production

Detection is a Poisson process. Its rate is built from affected traffic, not from the release strategy:

affected/hour = requests/hour × exposed fraction × manifest rate

rate = affected/hour × distinguishability × monitoring quality   (instruments)
     + affected/hour × reports per affected request                  (bug reports)

So the chance of catching a defect inside a canary soak of length S is 1 − e−rate×S. A canary enters only through the exposed fraction. That is the whole of its effect, and it is why the answer depends on the defect: at twenty thousand requests an hour a defect manifesting on 30% of them is caught in the soak essentially always, and one manifesting on 1 request in 100,000 essentially never. Nothing hard-codes that a canary improves the change failure rate.

The bug-report channel is proportional to affected users rather than a flat background rate, because a user can only report what they experienced. A flat rate would quietly guarantee that every defect is eventually found, which would delete the “exposed but never detected” case the model exists to show.

Blue-green verification uses the same defect attributes against a fixed number of synthetic requests, so it misses a rare defect for the same reason a canary does — not enough requests exercise it. What it does differently is expose nobody while it looks.

How the measures are computed

MeasureDefinition used here
Lead time for changes first commit to first arrival in production, including any rework. One observation per change: a rolled-back change shipped again does not produce a second, or the mean would depend on how often the factory fails.
Defects reaching production defects whose change served real traffic, counted whether or not anybody ever noticed. Split in the readout into those detected after release and those still live and undetected.
User exposure requests served by a defective release while it is live, integrated over the exposed traffic share. Reported as a total, a per-day rate, and the median per failed deployment — the last because the total is dominated by the long tail of slowly-detected defects, so a canary moves the median far more than the mean.
Deployment frequency releases that reached real users, per measured day. A blue-green release aborted at verification never reached users, so it is not a deployment.
Change failure rate failed deployments ÷ deployments — not changes. A deployment fails if a defect it carries is detected in production.
Time to restore service detection to rollback complete. The clock starts when somebody noticed, not when the defect was exposed.

Fixed parameters and reproducibility

The engine is a pure module published as window.ReleaseFactory with no reference to the page. It uses six independent seeded generators — arrivals, defect attributes, rework regressions, test verdicts, durations, production detection — and no unseeded random source and no clock, so the same seed and parameters replay the same event trace exactly, and switching one policy leaves the others’ draws where they were. Its test suite extracts this element from the published file, so the tested code and the code you are running are the same bytes. Every value the four primary controls hold fixed is listed, with its default, in Advanced controls.

Licence

Teaching materials © James C. Davis, licensed CC BY 4.0. One file, no dependencies, no network request; Download HTML hands you this same artifact.