Failure-Aware Software Development

What can we learn from the ways software systems fail?

An assumption fails in a real system. What happened is preserved as evidence: reports, artifacts, decisions and consequences. Comparing incidents identifies the failure mechanism — which assumption failed and under what conditions. Generalizing beyond the incident yields a reusable claim about how systems fail. Institutionalizing that claim changes requirements, design, analysis, validation, process and governance, which in turn changes the conditions under which the next system is built.

Software failures are investigated, patched, and filed away. The engineering knowledge they contain rarely survives the incident that produced it, so organizations meet the same class of failure again in the next system.

We study failures empirically and ask what would have to change for that knowledge to persist. Some of our work characterizes how failures recur across systems and domains; other work examines whether evidence from past failures actually changes engineering decisions, and how organizations might structure requirements, design, and validation so that it does.

Characterizing how software fails

Before failures can teach anything, we have to know what recurs. This work asks what failure studies actually establish, and whether the same engineering problems appear across systems and domains.

• Reflections on Software Failure Analysis
  ESEC/FSE-IVR · 2022
  Surveyed how failure studies are actually conducted and found the methods inconsistent, which is the problem the rest of this program addresses.
• Reflecting on Recurring Failures in IoT Development
  ASE-NIER · 2022
  Showed that IoT failures recur through a small number of design flaws, so the knowledge to prevent them existed before the failures happened.

Building evidence from failures

Individual failures are useful evidence only if they can be accumulated and analyzed. We study how to turn incident reports and other records of failure into evidence about recurring engineering problems.

• FAIL: Analyzing Software Failures from the News Using LLMs
  ASE · 2024
  FAIL scales failure analysis by extracting incidents from news reports, so the evidence base stops being limited to what one team can read.
• An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures
  SCORED · 2023
  Tests whether language models can do the reading that failure analysis requires, which is the bottleneck on doing it at scale.

Turning failure into engineering knowledge

Finding patterns is not enough. We study what can actually be learned from failures, and how that knowledge can be represented so it survives the incident that produced it.

• Learning From Software Failures: A Case Study at a National Space Research Center
  ICSE · 2026
  A case study inside a national space research center: what an organization with strong incentives to learn from failure actually does.
• On the Contents and Utility of IoT Cybersecurity Guidelines
  PACMSE · 2024
  IoT security guidelines are a written form of failure knowledge; this asks what they contain and whether they are usable by the engineers they target.

Getting knowledge back into engineering decisions

The final problem is closing the loop. Even good evidence has little value if it does not reach the people making decisions, or alter the structures through which engineering proceeds.

• Incorporating Failure Knowledge into Design Decisions for IoT Systems: A Controlled Experiment on Novices
  SERP4IoT · 2023
  A controlled study of whether failure knowledge changes design decisions. Moves the claim from plausible to tested.
• A Guide to Stakeholder Analysis for Cybersecurity Researchers
  arXiv · 2025
  Failure knowledge only changes engineering if it reaches the people who decide; stakeholder analysis is the missing step in most such work.
• Reflecting on the use of the Policy-Process-Product Theory in Empirical Software Engineering
  ESEC/FSE-IVR · 2023
  Examines the assumption underlying most of empirical software engineering — that policy and process shape product quality — and asks how often it is actually established rather than assumed.