Error breadcrumb storage: per-row error_context + generic OperationFailureEvent#

Context#

When a pipeline operation fails, the only durable record is str(exc) written to a free-text failure_error_message column (and the scattered unpack_failure_error_message / error_message siblings; RawDataPackage had no error field at all). For a bare FileNotFoundError that string is just [Errno 2] No such file or directory: '<path>' — no operation id, no step, no host, no side, no traceback, no “why”. The rich failure context that on_failure already assembles is sent to an email notification and then discarded, never persisted. And reset_state_on_failure (auto-retry) plus the frontend reset both NULL the message, so the trail of a flapping operation is erased exactly when it is most useful. This made staging failures cryptic and hard to debug.

Decision#

Store breadcrumbs in two places (hybrid):

  1. Per-row error_context (JSON) on each operation — the latest cause, for the UI’s current-error view. It mirrors the existing failure_error_message lifecycle: set on permanent failure, NULLed on reset. DataTransfer carries two (error_context + unpack_error_context) to match its two phases; RawDataPackage is brought up to parity.

  2. A generic OperationFailureEvent (a SystemLog polymorphic subclass) keyed by (operation_type, operation_id), appended on every failure (retryable and permanent) and never erased by a reset — the append-only durable trail.

error_context has a two-tier schema: Tier 1 (always available from the failure hook) and Tier 2 (the where/why — step, side, path, host, diagnosis — filled by boundary-wrapping, null until then). A capped traceback is stored always, not gated on DEVELOPMENT_MODE.

Write path: the base on_failure builds the error_context dict once (the notification formats from it) and appends the OperationFailureEvent uniformly; subclass hooks only write the dict to their model-specific column alongside their status transition.

Considered options#

  • Per-row JSON column only. Simplest, but loses the flap trail on reset/retry — the exact pain. Rejected as insufficient.

  • Per-operation log subclasses (DataTransferFailureEvent, …) with real FKs. Type-safe and matches DataTransferLog, but ~5 tables + boilerplate and awkward for the two-phase DataTransfer. Rejected for cost/complexity.

  • Hybrid (chosen). Cheap visible win now (per-row) + durable trail (generic event).

Consequences#

  • The per-row error_context is an intentional denormalized cache of the most recent OperationFailureEvent. The duplication is deliberate (cheap UI reads, minimal UI change).

  • OperationFailureEvent has no FK to the operation (it can’t — it spans operation types). It keys on (operation_type, operation_id), the pair the whole system already correlates on (Celery task state, recovery handlers, circuit breaker, routing). The missing FK follows existing precedent (PhysicalCopy.deletion_task_id is a bare Integer).

  • Split-neutral w.r.t. the DataTransfer transfer/unpack split (ops-db#84): the generic history table needs zero changes whichever way that goes; only the two per-row columns would later collapse to one per record.

Note: NUL-safety of the breadcrumb (#173)#

Breadcrumb persistence relies on ops-db’s NUL-safe JSONB_VARIANT column type: a U+0000 (NUL) in any string value is replaced by the sentinel ␀ (U+2400) at the type boundary on every write, all dialects. This matters because the highest-risk producer — a transfer exception whose message is raw subprocess stderr (BBCPError(message=result.stderr)) — can carry embedded NULs, and PostgreSQL jsonb rejects a NUL. data-transfer deliberately does not scrub locally: the fix is enforced once at the ops-db type boundary (ops-db ADR-0003), not in each of the ~28 writers. A regression test (tests/test_error_context.py::test_nul_in_error_context_is_sanitized_end_to_end, ops-db #110) pins both sinks (per-row error_context + OperationFailureEvent).