---
status: accepted
---

# Error breadcrumb storage: per-row `error_context` + generic `OperationFailureEvent`

## Context

When a pipeline operation fails, the only durable record is `str(exc)` written to a
free-text `failure_error_message` column (and the scattered `unpack_failure_error_message`
/ `error_message` siblings; `RawDataPackage` had no error field at all). For a bare
`FileNotFoundError` that string is just `[Errno 2] No such file or directory: '<path>'` —
no operation id, no step, no host, no side, no traceback, no "why". The rich failure
context that `on_failure` already assembles is sent to an email notification and then
**discarded**, never persisted. And `reset_state_on_failure` (auto-retry) plus the
frontend reset both NULL the message, so the trail of a flapping operation is erased
exactly when it is most useful. This made staging failures cryptic and hard to debug.

## Decision

Store breadcrumbs in **two** places (hybrid):

1. **Per-row `error_context` (JSON)** on each operation — the *latest* cause, for the
   UI's current-error view. It mirrors the existing `failure_error_message` lifecycle:
   set on permanent failure, NULLed on reset. `DataTransfer` carries two
   (`error_context` + `unpack_error_context`) to match its two phases;
   `RawDataPackage` is brought up to parity.
2. **A generic `OperationFailureEvent`** (a `SystemLog` polymorphic subclass) keyed by
   `(operation_type, operation_id)`, appended on **every** failure (retryable and
   permanent) and **never** erased by a reset — the append-only durable trail.

`error_context` has a two-tier schema: Tier 1 (always available from the failure hook)
and Tier 2 (the where/why — `step`, `side`, `path`, `host`, `diagnosis` — filled by
boundary-wrapping, `null` until then). A **capped** traceback is stored always, not gated
on `DEVELOPMENT_MODE`.

Write path: the base `on_failure` builds the `error_context` dict once (the notification
formats from it) and appends the `OperationFailureEvent` uniformly; subclass hooks only
write the dict to their model-specific column alongside their status transition.

## Considered options

- **Per-row JSON column only.** Simplest, but loses the flap trail on reset/retry — the
  exact pain. Rejected as insufficient.
- **Per-operation log subclasses** (`DataTransferFailureEvent`, …) with real FKs.
  Type-safe and matches `DataTransferLog`, but ~5 tables + boilerplate and awkward for the
  two-phase `DataTransfer`. Rejected for cost/complexity.
- **Hybrid (chosen).** Cheap visible win now (per-row) + durable trail (generic event).

## Consequences

- The per-row `error_context` is an intentional denormalized cache of the most recent
  `OperationFailureEvent`. The duplication is deliberate (cheap UI reads, minimal UI
  change).
- `OperationFailureEvent` has **no FK** to the operation (it can't — it spans operation
  types). It keys on `(operation_type, operation_id)`, the pair the whole system already
  correlates on (Celery task state, recovery handlers, circuit breaker, routing). The
  missing FK follows existing precedent (`PhysicalCopy.deletion_task_id` is a bare
  `Integer`).
- **Split-neutral** w.r.t. the `DataTransfer` transfer/unpack split (ops-db#84): the
  generic history table needs zero changes whichever way that goes; only the two per-row
  columns would later collapse to one per record.

## Note: NUL-safety of the breadcrumb (#173)

Breadcrumb persistence relies on ops-db's NUL-safe `JSONB_VARIANT` column type: a
U+0000 (NUL) in any string value is replaced by the sentinel `␀` (U+2400) at the
type boundary on every write, all dialects. This matters because the highest-risk
producer — a transfer exception whose `message` is raw subprocess stderr
(`BBCPError(message=result.stderr)`) — can carry embedded NULs, and PostgreSQL
`jsonb` rejects a NUL. data-transfer deliberately does **not** scrub locally: the
fix is enforced once at the ops-db type boundary (ops-db ADR-0003), not in each of
the ~28 writers. A regression test
(`tests/test_error_context.py::test_nul_in_error_context_is_sanitized_end_to_end`,
ops-db #110) pins both sinks (per-row `error_context` + `OperationFailureEvent`).