Uniform pipeline operation model#

Context#

“Stage” means three structurally different things across the pipeline today, so the system has no single handle on its own steps:

  • packaging status lives on the artifact — RawDataPackage.status

  • bundling status lives on the artifact — DataTransferPackage.status

  • transfer is a dedicated operation row — DataTransfer.status

  • unpack is a phase on another op’s row — DataTransfer.unpack_status (the transfer/unpack “unicorn record”: one row encoding two operations via paired columns status/unpack_status, retry_count/unpack_retry_count, error_context/unpack_error_context, …)

  • archive is a dedicated row using a differently named counter — LongTermArchiveTransfer.attempt_count (not retry_count)

  • staging is a single-phase StagingJob doing download+unpack as one all-or-nothing op

The shared Status enum (PENDING/SCHEDULED/IN_PROGRESS/COMPLETED/FAILED) is identical everywhere, but the usage has drifted into bugs: only bundling sets IN_PROGRESS; archive’s stuck-recovery resets healthy long-running transfers because it never sets it (#151); recovery is keyed by free-text strings that drift apart (recovery handler archive vs the long_term_archive the task writes → stalled archives never recover, #69; packaging never sets its kind at all). Counters are inconsistent (retry_count / unpack_retry_count / attempt_count; PhysicalCopy persists none — #90, #75).

Operator-chosen rollback (#149, “walk up the pipeline state chain”) is blocked on this: it needs every stage to be a uniform, independently addressable operation. This ADR establishes that model so #149 is purely additive.

Why now: the database is disposable (re-seed is cheap, no production data), so these are clean schema migrations with no backfill — the single largest risk is absent. That window closes when stable operations begin; this is the moment to pay the schema cost.

Decision#

Adopt one uniform operation model. Six coupled choices:

  1. One polymorphic Operation base table (joined-table inheritance), the same pattern as SystemLog / PhysicalCopy. Subclasses carry stage-specific fields: PackagingOperation, BundlingOperation, TransferOperation, UnpackOperation, ArchiveOperation, StagingOperation. Packaging and bundling are lifted off their artifacts: RawDataPackage / DataTransferPackage keep only their lifecycle state (PackageState), and operation status moves to the Operation row. The transfer/ unpack unicorn record splits into two sibling rows (this realizes ops-db#84, which this ADR subsumes).

  2. One canonical operation state machine. Every operation walks PENDING → SCHEDULED → IN_PROGRESS → COMPLETED/FAILED. IN_PROGRESS is mandatory and set in one place — the CCATEnhancedSQLAlchemyTask base (which already owns the start hook + heartbeat) — giving the DB an honest “running” signal and structurally fixing the #151 class for all stages. Packaging adopts the standard poll→SCHEDULED→dispatch find-work loop (dropping inline create-and-dispatch), making it re-runnable from PENDING and closing its double-dispatch race (#70-class). One retry_count on the base replaces attempt_count / unpack_retry_count and gives #75 a single get_retry_count().

  3. One OperationKind(str, Enum) whose values equal the frozen breadcrumb strings (raw_data_package, data_transfer_package, transfer, unpack, long_term_archive, staging). It is the single source of truth for the identity axis — recovery, circuit breaker, OperationFailureEvent key (ADR-0001), and the JTI discriminator all reference the enum member, never a literal. Because values equal already-stored strings, this is a code refactor, not a data migration, and it makes the #69 drift class unrepresentable. The existing stale literal (recovery’s archive) is reconciled to OperationKind.ARCHIVE (= long_term_archive) on adoption. The routing axis (OperationType.value + QUEUE_OPERATIONS_BY_LOCATION_TYPE) is a separate, frozen contract: queue names are built from it and workers self-discover queues at startup, so nothing here changes how workers run. Identity (operation_kind) and routing (operation_type) must stay distinct in code.

  4. Artifact/copy-anchored lineage. An Operation consumes input PhysicalCopy(s) and produces output PhysicalCopy(s); this produce/consume contract is identical for every kind. Lineage is traced through the artifact/copy graph (RawDataFile → RawDataPackage → DataTransferPackage), not through operation-to-operation FKs — because the pipeline’s fan-in (bundling) and fan-out (unpack) live between artifacts. Transfer and unpack are siblings on the same DataTransferPackage, not unpack-hung-off-transfer. This is the walk-up seam #149 builds on.

  5. OperationGroup for batched operations. A generic polymorphic parent that bundles N co-created operations and carries a status derived from its children (never set independently). StagingJob becomes OperationGroup plus a retention claim — its active flag holds the staged packages until released (deletion_manager already enforces this). The base is built minimal/claim-free now; its known future second customer is grouping the multiple results of a workflow reduction step.

  6. Deletion stays out, on shape grounds. Deletion is a PhysicalCopyStatus state machine on the artifact (PRESENT → DELETION_* → DELETED), running in the destroy direction — not a unit of work with a Status. Forcing it into the Operation model would fabricate an op row for an artifact transition. Its missing counter (#90) is fixed separately on PhysicalCopy.

Migration is additive and incremental: build the tables, migrate one stage at a time behind reconcile_package_state / classify_transferring_step, staging-first, leaning on the ops-db#78 Alembic drift gate. New/moved JSON columns land as jsonb, not json (ops-db#92).

Considered options#

  • Status stays on the artifact (no lift). Rejected: leaves packaging/bundling as snowflakes, so “every stage is an operation” stays false for two of five and #149 can’t address them uniformly. The act of packaging and the package artifact are genuinely separate concerns, and reconcile_package_state already derives state from operation statuses (ADR-0004, ops-db-api).

  • Separate per-stage tables sharing an OperationMixin (uniform shape, no base table). Lower blast radius and closest to today, but uniformity is by convention not schema, operation_id keeps colliding across tables (so OperationFailureEvent can never become a real FK), and #149’s “same handle” stays a code-level abstraction over heterogeneous tables. Rejected for a disposable-DB window where the stronger cut is affordable.

  • Operation-chained lineage (upstream_operation_id, unpack 1:1 off transfer). Rejected: a 1:1 chain can’t represent bundling fan-in / unpack fan-out, which live between artifacts.

  • Keep raw operation_type strings (no enum). Rejected: preserves the stringly-typed drift that produced #69 and the packaging mismatch.

  • New clean OperationKind enum + translation layer to the frozen strings. Rejected: the mapping boundary is a new drift surface that reintroduces the #69 class and forces a history consideration — strictly worse than values-equal-frozen-strings.

  • Merge the identity and routing axes into one enum. Deferred, not chosen: routing values feed dynamically-discovered queue names, so changing them would silently spin up new queues and orphan in-flight messages (a live-drain hazard) and force coordinated worker redeploys. Not required for uniformity.

  • Drop IN_PROGRESS, rely only on the Redis heartbeat. Rejected: loses the DB-visible running state the UI and stuck-detection need; the heartbeat stays the truth for liveness, the DB state is its legible projection.

  • Fold deletion into the Operation model. Rejected on shape (see Decision 6).

Consequences#

  • Cross-repo, lockstep. ops-db (models + migrations) → ops-db-api (reconcile_package_state, classify_transferring_step, attribution, reset endpoints — all read the duality directly today and must move together; ADR-0004 preserved) → data-transfer managers + base task → ops-db-ui. The UI is more coupled than first assumed: 8–12 components branch on operation kind and read paired fields and per-kind failure counts, so it does not “barely move.”

  • Realizes and supersedes ops-db#84; folds in #69, #75, the LTA counter; generalizes #151 (the per-stage IN_PROGRESS fix). #151 remains the immediate standalone fix.

  • Breadcrumbs (ADR-0001) stay correct. OperationKind values equal the frozen OperationFailureEvent.operation_type strings, so existing history stays correlatable; with a single global operation_id the loose (operation_type, operation_id) key may later become a real FK (a follow-up, not required here).

  • PackageState (state) is unchanged — it remains the artifact lifecycle, derived from operation statuses; it is not absorbed into Operation.status.

  • Enables #149 by exposing every stage as a uniform operation with consumed/produced PhysicalCopy lineage; the rollback verb itself stays out of scope.

  • No backfill, no production downtime because the DB is disposable — the deciding factor in choosing the deeper cut and in timing it before stable operations.