Appendix A - 16. Deploy heartbeats + stale-worker detection

The Structure of Deploy heartbeats + stale-worker detection — its shape at a glance:

The phase loop emits phase-plus-elapsed on a fixed cadence; a consumer that knows the cadence reads a missing beat as a hang rather than mistaking a slow phase for a dead one.

flowchart LR
  Loop[Deploy phase loop] -->|every 30s| Beat[/phase=X elapsed=Ns/]
  Beat --> Consumer{Beat within cadence?}
  Consumer -->|phase advancing| Live([Live and progressing])
  Consumer -->|no beat for N cycles| Stale([Flag as hung])

Accessible description: a deploy phase loop emits a phase-and-elapsed heartbeat every thirty seconds; a consumer that knows the cadence reads an advancing phase as live-and-progressing and a missing beat over several cycles as a hang to flag.

Projected from the catalogue entry lifecycle-and-observability / Deploy heartbeats + stale-worker detection.

On this page: Intent · Motivation · Applicability · Structure · Sample Code · Consequences · Example use within DocAble · Related Patterns

Intent

Intent — Periodic [heartbeat] phase=X elapsed=Ns emissions from long-running deploys, plus a stale-worker sweep, so a hung deploy or worker is distinguishable from a merely slow one.

Motivation

A long deploy (25–40 min) or a worker that stalls is indistinguishable from progress without a liveness signal: you cannot tell "slow build" from "hung process." The failure is blind waiting and undetected hangs: the orchestrator either kills a healthy-but-slow deploy or waits forever on a dead one. It recurs on every long-running operation.

Applicability

A phase loop with a periodic emit. The operation must be structured into observable phases.
A parseable heartbeat format (phase= / elapsed=) a consumer can grep.
A consumer that knows the expected cadence, so "no heartbeat for N×30 s" reads as stale.

Structure

The Structure diagram appears at the top of this page.

Sample Code

The heartbeat carries phase and elapsed, so both liveness and progress are observable; a consumer that knows the cadence flags a worker that has stopped beating. Process-alive is not enough — a deadlocked process is alive; only an advancing phase tells slow from hung.

import time

def deploy_with_heartbeat(phases, emit, interval=30.0):
    start = time.monotonic()
    for name, run_phase in phases:
        last = time.monotonic()
        for _ in run_phase():                          # phase yields progress ticks
            now = time.monotonic()
            if now - last >= interval:
                emit(f"[heartbeat] phase={name} elapsed={now - start:.0f}s")
                last = now

def is_stale(last_beat_ts: float, now: float, cadence=30.0, misses=3) -> bool:
    return (now - last_beat_ts) > cadence * misses     # no beat for N cadences ⇒ hung, not slow

Consequences

Liveness ≠ correctness. A deploy can heartbeat steadily while still failing — the signal proves it is moving, not that it is right.
Cadence knowledge is required. Judging staleness needs the consumer to know the 30 s cadence; without that the signal is just noise.
Granularity + noise. 30 s resolution can miss short hangs, and the heartbeat lines add log volume.

Example use within DocAble

The [heartbeat] phase=X elapsed=Ns emissions from the deploy phase loop.
The startup concurrency guard (refuses overlapping deploys).
The stale-worker detection sweep.

See also (sibling) — agent-registry, typed-event-bus: the fleet's other signal surfaces; heartbeats are the deploy-specific one.
Layer — supports staged-deploy-gates: it is the liveness signal over the deploy those gates run.