Stale-FAILED packages: an explicit “Inconsistent” bucket, reconciled from ops#

ops-db-ui #37 (Transfers dashboard rework). Builds on #114 (reset restores package state) and the data-transfer breadcrumb work (error_context).

Context#

attribute_failed_package charges a failed package to the latest currently-FAILED related op. A package in state=FAILED with no related op currently FAILED falls to a residual bucket — historically labelled unknown. On staging ~95 of ~105 failures were this bucket: the failure surface read ~90% “we don’t know where”.

These packages are not failing now — their lifecycle state is stale, left FAILED after an op was reset/recovered before #114 made reset also restore the package state. #114 prevents new occurrences; the existing pile is pre-#114 residue and mostly predates error_context, so the package row often carries no breadcrumb.

Calling this unknown is dishonest twice over: it reads as “unattributable mystery” when the real meaning is “state disagrees with the operations”, and re-deriving a pipeline position for a package that may have recovered would assert a live stuck-point that does not exist.

Decision#

Three coupled changes, all preserving the partition invariant (a package only moves between buckets, never disappears):

  1. Rename the bucket to inconsistent and surface it as its own explicit, explained group (“state=FAILED, no operation currently failed”), distinct from the genuine per-step failures — not as a funnel step.

  2. Breadcrumb-or-inconsistent attribution. In the residual branch, if the package carries an error_context with a step/operation_type, charge that step’s failed count and present it as “last failed at X” — never “stuck at X”. The breadcrumb is read server-side only to pick the cell; the error text is not shipped to the overview payload (consistent with ADR-0002). Packages with no breadcrumb stay in inconsistent.

  3. Reconcile, not flip. The admin action on an inconsistent package recomputes its state from its actual operation statuses — transfer+unpack+LTA all complete → ARCHIVED, mid-pipeline → TRANSFERRING, nothing started → WAITING. Idempotent. The one-off cleanup of the existing ~95 is this same bulk action run once, not a migration or script.

Considered options#

  • Reconcile from ops (chosen) correctly handles both sub-cases: a stale package that is actually fully archived becomes ARCHIVED; a mid-pipeline one becomes TRANSFERRING. Honest and self-correcting.

  • Flip to TRANSFERRING (mirror #114’s op-reset side effect) is less new code but sends already-complete packages bouncing back through the pipeline until a manager re-derives ARCHIVED. Rejected as misleading for the fully-archived sub-case.

  • Positional reconstruction (place a FAILED package at its first-incomplete phase regardless of breadcrumb) asserts a live position for packages that may have recovered. Rejected as dishonest.

  • Dismiss without state change hides the bad rows while leaving the DB inconsistent. Rejected.

Consequences#

  • unknown_failed_raw_packages / the unknown step label become inconsistent across ops-db-api and the ops-db-ui Failures panel; the reconciliation invariant is re-derived in those terms.

  • A new admin-gated reconcile endpoint (per-package and bulk) is added for RawDataPackage — the operation type the existing reset endpoints did not cover. Its actions are admin-only (ADR-0002 principle: Observers cannot act).

  • Honest surfacing is achieved by fixing state (reconcile + cleanup), not by relabelling — the bucket drains toward zero rather than being hidden.