Security Engineering
What can an adversary do, and which security properties must the system preserve?
Premise. A system that behaves correctly under ordinary use may fail when an adversary deliberately chooses inputs and actions intended to violate its requirements. Engineers must model the system and its adversaries, identify consequential threats, select controls, and obtain evidence that those controls preserve the required security properties.
Our campus reservation service now operates in a suitable environment, stores reservations persistently, and has been examined for architectural, performance, and reliability concerns. It authenticates students through the university identity provider and permits them to create and cancel reservations. But what happens when a student attempts to cancel someone else's reservation? What if an attacker deliberately exhausts the service's capacity, or causes a component to interpret untrusted data as an instruction?
Security engineering introduces adversarial behavior into the assumptions under which we design and evaluate software. In reliability engineering, we considered failures that might occur during operation. An adversary can deliberately select inputs, repeat operations, exploit knowledge of the system, and coordinate actions to produce an unwanted outcome. Some resulting failures resemble ordinary reliability failures, but the engineer must account for an actor actively seeking them.
Security is also an economic decision. The engineer cannot eliminate every conceivable threat at any cost. The value of the assets, the capabilities of plausible adversaries, the consequences of compromise, and the cost of protection all affect which controls are justified.
Structure of this unit¶
Lecture 1, Threat Modeling. We construct a system model of the reservation service, identify its assets and trust boundaries, and use STRIDE to examine how an adversary might violate its security objectives. To practice this, see Exercise 1.
Lecture 2, Controls & Assurance. We translate selected threats into security policies and enforceable controls. We examine authorization, least privilege, complete mediation, isolation, and security testing, then consider how unexpected authority arises in conventional software and agentic systems. To practice this, see Exercise 2.
The two lectures follow the Act III engineering method: identify the problem, construct a model, exercise judgment about the design, realize the selected controls, and obtain evidence that they satisfy the intended obligations.
Modeling the system and adversary¶
Security analysis begins with two questions. What must the system protect, and what can an adversary do?
An asset is something whose confidentiality, integrity, or availability matters to the system's stakeholders. The reservation service manages student identities, reservation records, and administrative permissions. Different assets have different security requirements. Students may be permitted to read room availability without being permitted to read the identities of everyone using a room. They may modify their own reservations but not those belonging to others.
An adversary model identifies the actors and capabilities relevant to the analysis. An unauthenticated attacker can submit requests to public endpoints. An authenticated student can invoke operations available to students and supply arbitrary request parameters. An administrator possesses additional permissions, while a compromised service account may exercise whatever authority its credentials provide.
The engineer must decide which capabilities to include. Assuming that an attacker can control the operating system produces a different analysis from assuming that the attacker can only submit HTTP requests. Neither assumption is universally correct; the model must reflect the system's intended environment and the threats the engineer is responsible for addressing.
| Model element | Engineering question | Reservation-service example |
|---|---|---|
| Asset | What must be protected? | Reservation ownership |
| Security objective | What property must hold? | Only authorized users may cancel reservations |
| Adversary | Who might violate the property? | Authenticated student |
| Capability | What can the adversary control? | HTTP requests and reservation identifiers |
| Attack surface | Where can the adversary interact with the system? | Reservation API |
| Trust boundary | Where must an assumption about authority or trust be established? | Between client requests and server-side operations |
The system model complements the adversary model. A data-flow diagram represents the actors, processes, data stores, and information flows relevant to the analysis. Trust boundaries identify places where information or requests move between components operating under different trust assumptions or authorities.
TB1 separates client-controlled requests from server-side execution. TB2 and TB3 represent boundaries involving external identity services and database authority. The engineer must establish the actual trust assumptions and permissions at each boundary. Note what the identity exchange does and does not say: the service receives an assertion the provider signed and verifies it against the provider's keys. It does not send the provider its own claims about who the user is.
A diagram is useful only if it preserves the information needed for the security decision. The engineer should refine a process when its internal structure affects the analysis, identify omitted data flows, and record assumptions about external services. A diagram that shows a web server and database but omits the identity provider cannot adequately explain how the service establishes user identity.
A trust boundary does not necessarily correspond to a machine boundary. Two components running on the same machine may possess different privileges, while separate machines may operate under a shared administrative authority. The relevant distinction concerns which assumptions and permissions are valid on each side.
Identifying threats¶
The engineer examines the system model to identify ways that an adversary could violate a security objective. The STRIDE framework provides six categories that help make this analysis systematic.
| STRIDE category | Reservation-service threat | Security concern |
|---|---|---|
| Spoofing | Attacker impersonates another student | Authentication |
| Tampering | Attacker modifies reservation data | Integrity |
| Repudiation | Student disputes having canceled a reservation | Accountability |
| Information disclosure | Student obtains another student's private information | Confidentiality |
| Denial of service | Attacker exhausts service capacity | Availability |
| Elevation of privilege | Student obtains administrative authority | Authorization |
STRIDE is a classification aid, not a complete threat model. Writing "elevation of privilege" does not explain how an attack occurs or what the engineer must prevent.
A useful threat statement identifies the actor, entry point, action, condition enabling the attack, and consequence. Consider:
An authenticated student submits a cancellation request containing another student's reservation identifier. If the API checks authentication but not ownership, the student can cancel a reservation they do not control.
The statement identifies a concrete security obligation: the server must establish that the requesting principal is authorized to cancel the specified reservation.
A threat is a possible source or course of harmful action. A vulnerability is a weakness that may permit the threat to succeed. An attack is an attempt to exploit such a weakness. These distinctions matter because a threat may remain relevant even after a particular vulnerability has been corrected.
Denial of service is the category students most often leave abstract, so the assigned Cloudflare postmortem supplies a concrete mechanism. A regular expression whose matching cost grows explosively on certain inputs lets a small request consume a large amount of CPU, and a service that accepts such input from anyone has handed an attacker a lever. Work through the expression before reading the outcome, and predict which inputs are expensive.
To practice this, see Lab 1.
The engineer should also examine the paths by which an attacker can reach consequential operations. For example, the reservation API may enforce ownership correctly while a separate administrative endpoint exposes the same operation without equivalent checks. The relevant model must include both paths.
Reviewing and prioritizing the model¶
A threat model is necessarily incomplete. The engineer must decide whether it is sufficiently complete for the decision at hand.
Review includes checking that the diagram matches the implementation, identifying omitted flows and dependencies, examining trust boundaries, and determining whether each important threat has been described concretely. Independent reviewers may discover assumptions that the original modeler overlooked.
Threats also differ in importance. A student obtaining another student's reservation identifier may be less consequential than a compromised administrative credential permitting modification of every reservation. The engineer considers the plausibility of exploitation, the potential impact, existing controls, and the cost of additional protection.
These judgments should be explicit. Assigning numerical risk scores does not eliminate uncertainty about attacker behavior or consequences.
Specifying security policy¶
Identifying a threat does not establish how the system should prevent it. The engineer must specify the behavior the system will permit and determine where that policy will be enforced.
For the reservation service, an authorization policy relates principals, resources, and operations. A student may cancel their own reservation, while an administrator may cancel any reservation. Authentication establishes the principal's identity; authorization determines whether that principal may perform the requested operation.
| Principal | Own reservation | Another student's reservation | Room administration |
|---|---|---|---|
| Student | Read, cancel | Denied | Denied |
| Administrator | Read, cancel | Read, cancel | Allowed |
| Service account | Only required internal operations | Only required internal operations | Denied unless explicitly required |
This matrix is a model of permitted behavior. The phrase "own reservation" describes a relationship between a principal and a resource, not merely a role assigned to the principal.
We can express the cancellation policy as a predicate:
MayCancel(u, r) = Admin(u) OR Owner(u, r)
The predicate is deliberately simple. A production policy might also consider reservation status, administrative scope, delegation, or institutional rules. Those additional conditions should be introduced because the requirements demand them, not merely because they are possible.
The engineer must determine which component evaluates this policy, how it obtains trustworthy information about the principal and resource, and whether every path to the protected operation applies the same rule.
Selecting and enforcing controls¶
A security control is a mechanism or procedure intended to prevent, detect, or limit a security violation. The engineer selects controls according to the threat model and the properties that must be preserved.
| Principle | Engineering decision | Reservation-service application |
|---|---|---|
| Least privilege | What authority does each component require? | Booking service cannot administer identities |
| Complete mediation | Where must access be checked? | Every cancellation requires authorization |
| Fail-safe defaults | What happens when permission is uncertain? | Deny the operation |
| Separation of privilege | Which actions require multiple conditions? | Administrative operation requires appropriate role and scope |
| Economy of mechanism | How can enforcement remain understandable? | Centralize authorization policy |
| Open design | What assumptions remain valid if the design is known? | Security does not depend on hiding API behavior |
These principles guide design but do not themselves enforce anything. The engineer must choose mechanisms with the necessary authority.
The policy must be enforced using trustworthy server-side information. A client-provided user identifier is not sufficient evidence of ownership. Likewise, validating that a reservation identifier is syntactically well formed does not establish that the requester is authorized to use it.
The engineer must also consider concurrency. If authorization depends on mutable information, a change between checking permission and performing the operation may invalidate the decision. The implementation must provide appropriate coordination between the check and the protected action, so that both observe the same state.
Isolation limits the consequences of compromise by separating components and restricting their capabilities. A booking service that cannot modify identity-provider records presents a smaller opportunity for abuse than one possessing unrestricted administrative credentials.
Multiple controls can protect the same obligation. This is commonly called defense in depth. The controls should address distinct failure paths rather than merely duplicate the same assumption.
When data acquires authority¶
Some security failures occur when a component treats information from a less-trusted source as instructions or grants it authority beyond what the engineer intended.
The Log4j vulnerability disclosed in December 2021 provides an example. Under affected configurations, attacker-controlled text processed by the logging framework could trigger lookup behavior with consequences beyond recording a message. The resulting vulnerability depended on the interaction among application input, logging functionality, external lookup mechanisms, and runtime configuration.
Agentic systems introduce a related engineering concern. An agent may retrieve a document containing instructions written by an untrusted party. If the agent treats those instructions as authoritative, the document may influence its subsequent use of privileged tools.
The mechanisms differ, but both raise questions about authority across trust boundaries. The engineer must determine which information is permitted to influence consequential operations and where that restriction can be enforced.
For conventional software, controls may include removing dangerous interpretation behavior, restricting outbound communication, and limiting process privileges. For agentic systems, controls may include restricting tool permissions, independently authorizing consequential operations, and separating retrieved content from instructions governing the agent.
Instructions asking an agent to ignore malicious content may be useful guidance, but they are not a substitute for enforcing tool permissions at the relevant boundary.
Obtaining security assurance¶
A security control is useful only if it provides the intended protection under the assumptions of the threat model. The engineer must therefore obtain evidence about both the control's implementation and its effectiveness.
| Security claim | Control | Evidence |
|---|---|---|
| Students cannot cancel others' reservations | Server-side authorization | Negative authorization tests |
| User input cannot alter SQL query structure | Parameterized queries | Code review and injection tests |
| A compromised service has limited authority | Restricted service credentials | Permission inspection and isolation tests |
| Known vulnerable dependencies are addressed | Dependency update and admission policy | Inventory and vulnerability checks |
| Untrusted documents cannot authorize privileged agent actions | Independent tool authorization | Adversarial tool-use tests |
Security assurance uses several complementary techniques. Static analysis examines program representations without executing the application. Dynamic analysis examines behavior during execution. Fuzzing supplies generated or mutated inputs to discover unexpected behavior. Penetration testing evaluates selected attack paths under an adversarial perspective.
These techniques produce evidence; they are not substitutes for the security controls they examine.
For example, fuzzing may reveal that an API accepts malformed requests, but the mitigation is a change to the implementation or its enforcement mechanisms. Similarly, a passing authorization test establishes behavior for the tested principals, resources, and operations. It does not establish that every possible authorization path has been examined.
The engineer should connect each consequential security claim to its enforcement mechanism and the evidence used to evaluate it.
Engineering judgment¶
Security engineering extends the Act III method to systems whose inputs and operating conditions may be chosen by adversaries. The engineer models the system and its adversaries, identifies threats to important assets, and specifies the security properties that must be preserved.
The threat model supports decisions about which risks require attention. The security policy specifies permitted behavior. Controls give selected obligations authority in the implementation, while security analysis and testing provide evidence about whether those controls work.
The engineer must also decide which risks remain acceptable. No finite threat model or test suite establishes that a system is secure against every possible adversary. The relevant question is whether the engineer can justify the chosen protections under explicit assumptions about the system, its environment, and its likely attackers.
The next unit examines how operators observe and maintain these properties after deployment, including how they detect, investigate, and respond when the system's behavior departs from expectations.
Labs¶
Lab 1 — Regular Expression Denial of Service¶
The instrument for the Cloudflare prediction. Four curated expressions run against generated inputs of increasing length in the browser's own engine, and the page reports measured execution time rather than a plotted complexity curve; you set the longest input to try, pick a linear or logarithmic time axis, and run the experiments singly or all four. They form a two by two. Experiments 1 and 2 run the same expression on different inputs, so the only thing that changes is what the adversary supplies, and experiments 2 and 4 run different expressions on byte-identical input, so the only thing that changes is what the engineer wrote. Watch where a run stops, because the page holds a wall-clock budget the matching thread cannot block and destroys the worker when a match overruns it: isolating the match bounds the damage, not the cost. The demonstration illustrates the class of algorithmic vulnerability rather than reproducing Cloudflare's production behavior, since regex engines differ in matching algorithm and optimizations.
Exercises¶
Exercise 1 — Threat-modeling reservation cancellation¶
The reservation service exposes an endpoint that cancels a reservation by identifier. Students authenticate through the university identity provider, and administrators may cancel reservations on behalf of students. Using the system model, students should:
- Identify the assets and trust boundaries relevant to cancellation.
- Describe three concrete threats, including the adversary, entry point, action, and consequence.
- Select the most consequential threat and explain what additional information would help evaluate it.
The exercise should produce a revised diagram and a short threat table, not merely a list of STRIDE categories. A second pass perturbs one assumption — the administrative interface is a separate service, the identity provider is briefly unavailable, reservation identifiers are sequential integers — and asks which threats that changes.
Exercise 2 — Securing reservation cancellation¶
Students return to the threat identified in Lecture 1: an authenticated student attempts to cancel another student's reservation, and administrators cancel on students' behalf through a second interface. The engineer must decide how to enforce the authorization policy and what evidence is sufficient to accept the implementation. Students should develop:
- An authorization rule identifying the principal, resource, operation, and required relationship.
- An enforcement design showing where the rule is evaluated and what happens when authorization fails.
- A test plan covering authorized operations, unauthorized operations, and attempts to bypass the enforcement path.
Does the proposed design apply the correct policy to both interfaces, or has it protected only the ordinary student endpoint? The intended outcome is a small, explicit security model connected to enforceable behavior and corresponding evidence.
Materials¶
- Lecture 1 slides — Security Engineering: Threat Modeling — source (PPTX)
- Lecture 2 slides — Security Engineering: Controls and Assurance — source (PPTX)
Readings¶
The method, from the people who had to run it
- Shostack, "Experiences Threat Modeling at Microsoft" (2008). Read the whole paper; it is eleven pages. This is where Lecture 1's method comes from: the data-flow diagram as the system model, and STRIDE applied per element rather than to the system as a whole. Two things in it carry the lecture. The author owned the method across a decade of mandatory use, so the limitations are stated by the person responsible for them rather than by a critic. And the validation heuristics in section 5.1 are the part students skip: a diagram nobody checked against the implementation describes a system that may not exist. The Microsoft-specific tooling and organizational history are context rather than content. Full citation: Adam Shostack, “Experiences Threat Modeling at Microsoft,” Proceedings of the Workshop on Modeling Security (MODSEC08), co-located with the International Conference on Model Driven Engineering Languages and Systems (MODELS) (Toulouse, France), CEUR Workshop Proceedings, vol. 413 (2008), https://shostack.org/files/papers/modsec08/Shostack-ModSec08-Experiences-Threat-Modeling-At-Microsoft.pdf.
Design principles that have not needed revision
- Saltzer and Schroeder, "The Protection of Information in Computer Systems", Proceedings of the IEEE 63, no. 9 (1975). Read the design-principles discussion in Section I, a few pages. The survey of 1970s protection hardware that follows is not assigned. The six principles in this unit's controls table are theirs, in their words, and they are worth meeting in the original because each one is an engineering question rather than a slogan. Least privilege asks what authority a component actually requires. Complete mediation asks where access is checked and whether any path avoids the check. Fail-safe defaults asks what the system does when permission is uncertain. The paper predates the web by two decades. Full citation: Jerome H. Saltzer and Michael D. Schroeder, “The Protection of Information in Computer Systems,” Proceedings of the IEEE 63, no. 9 (1975): 1278–308, the design-principles discussion in Section I.
What verification a developer owes before shipping
- Black, Guttman and Okun, NISTIR 8397, Guidelines on Minimum Standards for Developer Verification of Software (NIST, 2021). Read section 2, which states eleven recommended techniques and what each is for, including threat modeling, static analysis, fuzzing, hardcoded-secret review and checking included components. Section 3 is supplemental reference material to consult rather than read; section 4 goes beyond this unit. Read it as a statement of minimums rather than a recipe. The document is careful that each technique produces evidence about the property it examines and nothing else, which is the distinction behind this unit's claim, control and evidence table. Full citation: Paul E. Black et al., Guidelines on Minimum Standards for Developer Verification of Software, NISTIR no. 8397 (National Institute of Standards and Technology, 2021), https://doi.org/10.6028/NIST.IR.8397, section 2, the eleven recommendations.
One incident, four engineering questions
- Graham-Cumming, "Details of the Cloudflare Outage on July 2, 2019" (Cloudflare, 2019). Required. A single firewall rule carrying a regular expression with pathological backtracking saturated CPU across the network and took service down for 27 minutes. Note carefully what this is and is not. The rule was deployed internally, so the incident was not an attack. The algorithmic weakness it exposed is exploitable as a denial-of-service vulnerability all the same, which is what makes the postmortem a security reading and not only a reliability one. Four questions come out of the one incident. What property of the expression made certain inputs expensive? How could an attacker reach the same behavior deliberately? Why did the rule reach production so broadly? What test or control would have caught it beforehand? The postmortem also separates the defect from the organizational conditions that allowed it to cause an outage, which is the distinction the Reliability unit drew between a failure and the conditions that let it propagate. Full citation: John Graham-Cumming, “Details of the Cloudflare Outage on July 2, 2019,” Cloudflare, July 12, 2019, https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/.
Further depth
- Sommerville, Software Engineering, 9th edition, chapter 14, "Security engineering". Textbook framing for the unit: security risk management, design for security, and system survivability. Read 14.1 and 14.2. One caution about editions. The scan distributed with this course is the 9th edition, where security engineering is chapter 14; the 10th edition renumbers the subject to chapter 13 and reorganizes its sections, so a section reference taken from one edition will not locate the same material in the other. Full citation: Ian Sommerville, Software Engineering, 9th ed. (Addison-Wesley, 2011), chapter 14, sections 14.1 and 14.2.
- Anderson, Security Engineering: A Guide to Building Dependable Distributed Systems, 3rd edition, chapter 27, "Secure Systems Development" (Wiley, 2020; the author makes the chapters freely available). Optional, and long at forty-nine pages. Read it for the argument that security requirements emerge from a system and its environment rather than from a checklist, and for its treatment of risk management as a judgment made under uncertainty rather than a score to compute. Anderson writes from decades of deployments that failed, so the chapter is unusually direct about which practices survive contact with a real project. Full citation: Ross Anderson, Security Engineering: A Guide to Building Dependable Distributed Systems, 3rd ed. (Wiley, 2020), https://www.cl.cam.ac.uk/~rja14/book.html, chapter 27, Secure Systems Development.
- Shacham, "The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86)", ACM CCS 2007. Advanced and fabulous. Optional, and offered as an invitation rather than as material you are expected to master. The paper shows that an attacker who controls the execution path can assemble arbitrary computation from short instruction sequences already present in executable memory, injecting no code at all. Read it against the Cloudflare postmortem. One exploits the computational complexity of an ordinary operation; the other repurposes instructions that are individually legitimate. Both make the same point about trust boundaries and authority: what matters is not whether each piece is permitted on its own, but who controls how the pieces compose. Full citation: Hovav Shacham, “The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls (on the x86),” Proceedings of the 14th ACM Conference on Computer and Communications Security (CCS ’07) (Alexandria, VA), 2007, 552–61, https://doi.org/10.1145/1315245.1315313.
The Sommerville chapter frames the unit; the other two are optional.