Operations & SRE
Premise. A running system produces evidence that no predeployment activity can obtain. Operators must decide what to observe, what to do when the system misbehaves, and what to change once it recovers.
The reservation service is deployed, and students are booking rooms with it. Nothing in the previous units establishes whether it is working, because each of them reasoned about the system before it met its users. The tests ran against a workload we chose. The threat model described adversaries we imagined. Operations starts where those stop: behavior is now a fact rather than a prediction, and someone is accountable for it.
Structure of this unit¶
One lecture, run as an incident-response workshop rather than an exposition. You will receive incomplete information about a running service, decide what to do, then revise that decision as new evidence arrives. Four scenarios progress from stated alternatives to a response you propose yourself.
Analyze the reservation service during each incident, then apply one selected lesson to your own project after each half. Four incidents times two analyses does not fit in seventy-five minutes, and the second pass is the one that gets rushed. Projects differ in architecture, dependencies, data sensitivity and cost, so they do not owe their users the same operational guarantees, and the project exercises are where that transfer happens.
Keep an incident worksheet for each scenario, with six fields.
- What do we know? Observed facts, separated from inference.
- What do we suspect? Hypotheses, named as such.
- What evidence do we need? The next observation.
- What action will we take? The decision, and whose it is.
- What would change our mind? The revision condition.
- What should we change permanently? The durable control, added at the end.
The fifth field does the most work during a scenario: a decision recorded without its revision condition cannot be revisited honestly. The sixth is filled in once the incident is over, and it is the one we collect.
Every scenario sits somewhere on one loop.
Defining service health¶
A service level indicator measures one aspect of observed behavior. A service level objective sets a target for an indicator over a stated interval. Both are choices, and the choosing is where the engineering happens.
Infrastructure measurements do not establish health. Processor utilization, memory consumption and the proportion of HTTP 200 responses can all look unremarkable while the service fails at its purpose, because none of them mentions a reservation. An indicator earns its place by naming behavior a user would recognize: a confirmation that arrives, a booking that still exists tomorrow.
An error budget converts an objective into a quantity you can spend, and one percentage denotes two different quantities. Under a time-based availability measure, 99.9% over thirty days permits 43.2 minutes of unavailable time. Under a request-based indicator, it permits 0.1% of eligible requests to fail, which need not correspond to 43.2 minutes of anything. Decide which you mean before quoting the figure.
Some obligations resist the percentage form. No confirmed reservation is lost states an integrity obligation, not an availability target: a budget permitting 0.1% of confirmed bookings to vanish is not a service level anyone would accept. Telling the two apart is worth the argument it provokes.
Obtaining operational evidence¶
Three kinds of telemetry answer three different questions.
| Evidence | What it answers |
|---|---|
| Metrics | How much? How often? How widespread? |
| Logs | What event occurred, with what context? |
| Traces | Where did a particular request spend time? |
None substitutes for another. Metrics establish scope and trend and seldom explain a cause. Logs carry context about particular events and will not say how many users were affected. A trace follows one request across components and attributes its elapsed time, which makes it the only one that answers where did the time go directly.
Evidence also costs. Instrumentation consumes capacity, storage and attention, so nobody observes everything. An operator decides in advance which behavior to make visible, and that decision settles which incidents will be diagnosable at all.
Responding to incidents¶
An alert claims that a person should act now. Alerting on a symptom users feel keeps the claim honest. Alerting on every threshold crossing trains people to dismiss it.
A runbook states the symptoms, the diagnostic checks, the permitted actions, the escalation conditions, and how to verify that the service has recovered. It records decisions the organization made in advance, while leaving room for judgment when the incident differs from expectations. Followed mechanically through a situation it does not describe, a runbook is worse than none.
Mitigations trade one obligation against another, rarely symmetrically. Stopping writes, failing requests explicitly, queueing work for later, failing over, and continuing to retry each cost something specific. Queueing is the instructive case. Accepting a reservation request you cannot complete may be unsafe if the user believes a queued request is a confirmed booking, so failing explicitly may be preferable to accepting work whose outcome cannot be guaranteed. Timing is the other trap: an event that coincides with the onset of an incident is not thereby its cause, and a mitigation aimed at the wrong cause spends time the users are paying for.
Learning from incidents¶
Recovery ends the incident. It does not change the conditions that produced it.
A useful postmortem records the impact and timeline, the technical and organizational conditions that contributed, how the problem was detected and handled, the corrective actions with named owners, and the evidence required to close each one. Avoid reducing it to a root cause. A complex incident usually involves several contributing conditions, and the single-cause account appeals because it is short rather than because it is true.
Corrective actions differ in reach. Repairing a defect addresses that defect. A regression test addresses its recurrence. A new testing method addresses defects of the same kind. An alert shortens detection. A revised runbook improves the response. An architectural limit bounds the damage. Only some alter the conditions under which a similar incident can happen again, and separating the local repair from the durable change is the judgment this unit asks you to exercise.
Engineering judgment¶
Operational responsibility is continuous, which sets it apart from the activities in earlier units. There is no point at which the system has been operated. What an operator can know is bounded by what someone chose to observe, and every judgment is made while users are being served.
This returns to Failure-Aware Engineering. An observed failure should change the software and the mechanisms that govern future work on it, and repairing one failure is not the same as changing the conditions that let its class recur. The concluding unit asks that question of the course as a whole: which models an organization maintains, what evidence it demands, and which obligations justify enforcement.
Exercises¶
Exercise 1 — Define health for your own project¶
Three minutes, two on the worksheet and one on report-back from two teams. Define one indicator and one objective for your own project, then say what could go wrong while that indicator stays green. Write it in three lines — the indicator, named so a user would recognize the behavior; the objective, with its target, its interval, and the events that count as eligible; and the blind spot, the failure that keeps the indicator green.
Your project does not owe what the next team's owes. Where personal data is held, integrity and confidentiality may outrank availability. Where inference is paid for, cost per request can be the indicator that matters. Where the information is public, staleness may be the only failure users feel.
Exercise 2 — What would expose your project's failure? (optional)¶
Optional, and not run in the lecture. This one asks for more engineering than a classroom discussion holds: you have to identify a hard failure, reason about its candidate causes, find the observation that discriminates between them, and then specify instrumentation somebody could actually build. Take it as a project exercise in your own time, where it is worth the hour it deserves.
Name the hardest problem to diagnose in your project, then the one observation that would expose it. The pair is the answer: the failure stated specifically enough that two teams could disagree about its cause, and the observation stated specifically enough that somebody could instrument it tomorrow. Four starting points if your team is stuck — a dependency you do not own gets slower and nothing in your code changed; one user's request is slow and the aggregate never moves; the cost per request doubles and the latency does not; it happens only under load you have never generated.
Record both. An observation is a mechanism for producing evidence; it becomes a control once something reads it and acts, so name the acting half too.
Exercise 3 — Your project's critical dependency¶
Which dependency could disable your project, and what behavior would you permit while it is unavailable? Harder than listing your dependencies, because the second half is a policy. Write three lines — the dependency whose absence stops a consequential operation, which need not be one that stops you serving users at all; the permitted behavior, stated as a rule, for what the system may do while it is gone; and the tempting action you are ruling out in advance.
Three minutes, and two teams read all three lines rather than the first. The third line is the runbook's permitted-actions row, and it is the only one still useful at three in the morning.
Exercise 4 — What must change permanently?¶
Choose a failure your team has encountered. Propose a control that would prevent, detect, or limit similar failures in the future. The distinction is not control-versus-code — a durable control is often implemented in code, and a code fix can be part of one. It is between repairing the one instance and preventing, detecting or limiting a recurrence. Four lines, and the last two are the test — the failure, one you have actually had rather than one you fear; the control, which may be a lint, a gate, a test, an alert, or an architectural limit; the owner, a named accountable person; and the closing evidence, what someone could check to call it done.
The test of reach is four-way: would your control prevent the failure, detect it, limit its consequences, or improve recovery from it? One control per team, and this is the artifact the unit collects.
Materials¶
- Lecture slides — Operations and SRE — source (PPTX)
Readings¶
Deciding what counts as healthy
- Beyer, Jones, Petoff and Murphy (eds.), Site Reliability Engineering: How Google Runs Production Systems (O'Reilly, 2016), chapter 4, "Service Level Objectives". Read the whole chapter; the publisher makes the book freely available at sre.google/books. It separates the indicator from the objective and treats the choice of indicator as an engineering decision rather than a reporting convenience. Two things in it carry the lecture. An objective is stated over an interval and against a defined set of eligible events, which is what makes an error budget a quantity you can spend rather than a slogan. And an aggregate measure can look healthy while a subset of users is served badly, so the chapter argues for distributions over means. The account of Google's internal review process is context rather than content. Full citation: Betsy Beyer et al., eds., Site Reliability Engineering: How Google Runs Production Systems (O’Reilly Media, 2016), chapter 4, Service Level Objectives.
Turning an incident into organizational learning
- Beyer, Jones, Petoff and Murphy (eds.), Site Reliability Engineering (O'Reilly, 2016), chapter 15, "Postmortem Culture: Learning from Failure". Read the whole chapter. This is the more important of the two readings. It explains how an incident becomes organizational learning instead of an occasion to identify whoever made a mistake, and it ties each corrective action to an owner and to the evidence that closes it. Read it against the Failure-Aware Engineering unit: the argument is the one this course makes about converting an observed failure into a durable control. The chapter's templates and tooling matter less than its insistence that the record exists to be acted on. Full citation: Betsy Beyer et al., eds., Site Reliability Engineering: How Google Runs Production Systems (O’Reilly Media, 2016), chapter 15, Postmortem Culture: Learning from Failure.
If you read only one of these, read the second.