Split archiving into awaiting_archive + archiving: a seven-step funnel#

ops-db-ui #37 (Transfers dashboard rework). Amends epic #112, which established the six-step package funnel and its partition invariant.

Context#

Epic #112 derived a single funnel over RawDataPackage, partitioning the package total across six steps packaging · waiting · transfer · unpack · archiving · archived with the invariant Σ(occupancy + failed) + unknown == total_raw_packages.

classify_transferring_step returns archiving for two physically different populations: a package whose transfer + unpack are complete but which has no long-term-archive op yet (at rest, queued for the archive_manager), and a package with an LTA op actively in progress. On real staging data the first population dominated — ~3,013 packages sat in archiving while almost none were truly in flight — so “3,013 Archiving” read to operators as “3,013 moving”, a false sense of backlog activity. The at-rest backlog is the operationally interesting signal and it was invisible.

Decision#

Split the lumped cell into two ordered steps, making the funnel seven steps:

packaging · waiting · transfer · unpack · awaiting_archive · archiving · archived

  • awaiting_archive — TRANSFERRING, transfer + unpack complete, no LTA op exists (if not ltas). At rest; the archive_manager has not picked it up.

  • archiving — TRANSFERRING, transfer + unpack complete, an LTA op exists and is in progress. Actively archiving.

awaiting_archive has no native failure path — a package with no LTA op cannot have a failed LTA op — so its failed count is structurally 0 (an LTA failure charges archiving, as before). The invariant becomes Σ(occupancy + failed over 7 steps) + inconsistent == total_raw_packages (inconsistent is the renamed unknown; see ADR-0004).

The split reads pkg.long_term_archive_transfers, already selectinloaded for the funnel packages, so it adds no query to the overview hot path (no bearing on the #84 query-count work).

Considered options#

  • (a) Seven-step split (chosen). Honest at-rest vs in-flight separation in the funnel itself. Cost: reopens #112’s six-step contract — package_funnel array length, _FAILED_STEP_TO_FUNNEL_CELL, build_package_funnel, the reconciliation tests, and the ops-db-ui FUNNEL_META + a seven-card layout all change together.

  • (b) Additive sub-breakdown. Keep six steps; report archiving_awaiting / archiving_active as sub-counts the UI renders inside the Archiving card. Non-breaching and cheaper, but the at-rest backlog stays a second-class annotation rather than a first-class pipeline position. Rejected: the backlog is a primary operational state and deserves its own cell.

  • (c) Tooltip only. Explain the lump in the card tooltip, no data split. Rejected: leaves the misleading single number on screen.

Consequences#

  • package_funnel is now seven cells. Any consumer indexing it positionally or asserting “six steps” must update; consumers that match by step name and iterate keep working (ops-db-ui already drives cards from FUNNEL_META matched by step).

  • The reconciliation tests (test_overview_funnel.py) and the funnel↔Failures-panel reconciliation must be re-derived for seven steps.

  • The contract is again breaking-on-change: renaming/reordering steps or altering the invariant ripples across ops-db-api and ops-db-ui together, as it did for #112.