Appendix C - 11. Fuzz campaigns (+ auto-coverage)

The Structure of Fuzz campaigns (+ auto-coverage) — its shape at a glance:

The campaign generates malformed inputs and runs the tool against them; coverage is collected automatically and compared to a baseline. A finding routes to a fix aimed at the spec point, not the individual seed.

flowchart LR
  Gen[Malformed-input generator] -->|adversarial bytes| Run[Run tool]
  Run -->|coverage| Base[(Coverage baseline)]
  Run -->|crash / corruption| Fix["Fix to the spec point<br/>(not the seed)"]

Accessible description: a generator feeds adversarial bytes to the tool; coverage is collected and compared against a baseline so reach is measurable. A crash or corruption routes to a fix aimed at the stable spec point rather than the individual failing seed.

Projected from the catalogue entry regression-tests / Fuzz campaigns (+ auto-coverage).

On this page: Intent · Motivation · Applicability · Structure · Sample Code · Consequences · Example use within DocAble · Related Patterns

Intent

Intent — Campaigns that feed malformed and adversarial inputs to the tool to find crashes and corruption, with coverage collected automatically, plus an RCA discipline that fixes to the spec, not the failing seed.

Motivation

Real-world documents are malformed in ways no hand-written test anticipates: a truncated stream, an odd encoding, a structure right at the edge of what the spec allows. A fuzzer finds the crash or corruption on inputs you would never think to write. The failure is crashes / corruption on adversarial or spec-edge inputs, and it hides across an input space far too large to enumerate.

Applicability

Fuzz harnesses and a corpus/generator of malformed inputs.
A coverage collector + aggregator + baseline so reach is measurable.
RCA discipline (fix to the spec, not the seed) — without it, fuzzing devolves into seed whack-a-mole.

Structure

The Structure diagram appears at the top of this page.

Sample Code

The campaign loop is small; the discipline is what matters. Generate malformed inputs, run the tool, record any crash. On a finding, the fix targets the spec rule the input violated — so the same fix passes every input the spec allows, not just the one seed. Fixing the seed alone leaves the class open.

import random

def campaign(run_tool, seed_corpus: list[bytes], iterations: int = 10_000) -> list[bytes]:
    """Feed mutated bytes to the tool; collect inputs that crash or corrupt.
    Each finding is a *class* representative — fix to the spec rule it broke,
    not to this exact byte string, or the class stays open."""
    findings = []
    rng = random.Random(0)
    for _ in range(iterations):
        base = rng.choice(seed_corpus)
        mutant = bytearray(base)
        i = rng.randrange(len(mutant))
        mutant[i] ^= 1 << rng.randrange(8)   # flip one bit — cheap adversarial mutation
        try:
            run_tool(bytes(mutant))
        except Exception:
            findings.append(bytes(mutant))   # a crash on spec-allowed input is a real bug
    return findings

Consequences

Compute-heavy. Campaigns cost real time; coverage is tracked to know when they've saturated.
Baseline maintenance. The coverage baseline must be re-based only on intentional coverage-shape changes.
Seed-fixing is the anti-pattern the RCA discipline exists to prevent — fixing only the failing input leaves the spec-class open.

Example use within DocAble

The *Fuzz* / *Campaign* harnesses + auto-coverage collection and aggregation.
The fix-to-the-stable-spec-point RCA discipline.

See also (sibling) — property-tests (structured generation), test-onion-tiers (example tiers).
Counterpart — the coverage baseline tracks what the campaign reached, keeping "we fuzzed it" honest.