---
status: accepted
---

# Stale-FAILED packages: an explicit "Inconsistent" bucket, reconciled from ops

ops-db-ui #37 (Transfers dashboard rework). Builds on #114 (reset restores package
state) and the data-transfer breadcrumb work (`error_context`).

## Context

`attribute_failed_package` charges a failed package to the latest currently-FAILED
related op. A package in `state=FAILED` with **no** related op currently FAILED falls
to a residual bucket — historically labelled `unknown`. On staging ~95 of ~105
failures were this bucket: the failure surface read ~90% "we don't know where".

These packages are not failing *now* — their lifecycle state is **stale**, left
`FAILED` after an op was reset/recovered before #114 made reset also restore the
package state. #114 prevents new occurrences; the existing pile is pre-#114 residue
and mostly predates `error_context`, so the package row often carries no breadcrumb.

Calling this `unknown` is dishonest twice over: it reads as "unattributable mystery"
when the real meaning is "state disagrees with the operations", and re-deriving a
pipeline position for a package that may have *recovered* would assert a live
stuck-point that does not exist.

## Decision

Three coupled changes, all preserving the partition invariant (a package only moves
between buckets, never disappears):

1. **Rename the bucket to `inconsistent`** and surface it as its own explicit,
   explained group ("state=FAILED, no operation currently failed"), distinct from the
   genuine per-step failures — not as a funnel step.

2. **Breadcrumb-or-inconsistent attribution.** In the residual branch, if the package
   carries an `error_context` with a `step`/`operation_type`, charge that step's
   *failed* count and present it as **"last failed at X"** — never "stuck at X". The
   breadcrumb is read **server-side only** to pick the cell; the error text is not
   shipped to the overview payload (consistent with ADR-0002). Packages with no
   breadcrumb stay in `inconsistent`.

3. **Reconcile, not flip.** The admin action on an inconsistent package **recomputes
   its state from its actual operation statuses** — transfer+unpack+LTA all complete →
   `ARCHIVED`, mid-pipeline → `TRANSFERRING`, nothing started → `WAITING`. Idempotent.
   The one-off cleanup of the existing ~95 is this same bulk action run once, not a
   migration or script.

## Considered options

- **Reconcile from ops (chosen)** correctly handles both sub-cases: a stale package
  that is actually fully archived becomes `ARCHIVED`; a mid-pipeline one becomes
  `TRANSFERRING`. Honest and self-correcting.
- **Flip to `TRANSFERRING`** (mirror #114's op-reset side effect) is less new code but
  sends already-complete packages bouncing back through the pipeline until a manager
  re-derives `ARCHIVED`. Rejected as misleading for the fully-archived sub-case.
- **Positional reconstruction** (place a FAILED package at its first-incomplete phase
  regardless of breadcrumb) asserts a live position for packages that may have
  recovered. Rejected as dishonest.
- **Dismiss without state change** hides the bad rows while leaving the DB
  inconsistent. Rejected.

## Consequences

- `unknown_failed_raw_packages` / the `unknown` step label become `inconsistent`
  across ops-db-api and the ops-db-ui Failures panel; the reconciliation invariant is
  re-derived in those terms.
- A new admin-gated reconcile endpoint (per-package and bulk) is added for
  `RawDataPackage` — the operation type the existing reset endpoints did not cover.
  Its actions are admin-only (ADR-0002 principle: Observers cannot act).
- Honest surfacing is achieved by *fixing* state (reconcile + cleanup), not by
  relabelling — the bucket drains toward zero rather than being hidden.