---
status: accepted
---

# Uniform pipeline operation model

## Context

"Stage" means three structurally different things across the pipeline today, so the system
has no single handle on its own steps:

- **packaging** status lives on the artifact — `RawDataPackage.status`
- **bundling** status lives on the artifact — `DataTransferPackage.status`
- **transfer** is a dedicated operation row — `DataTransfer.status`
- **unpack** is a *phase on another op's row* — `DataTransfer.unpack_status` (the
  transfer/unpack "unicorn record": one row encoding two operations via paired columns
  `status`/`unpack_status`, `retry_count`/`unpack_retry_count`,
  `error_context`/`unpack_error_context`, …)
- **archive** is a dedicated row using a *differently named* counter —
  `LongTermArchiveTransfer.attempt_count` (not `retry_count`)
- **staging** is a single-phase `StagingJob` doing download+unpack as one all-or-nothing op

The shared `Status` enum (PENDING/SCHEDULED/IN_PROGRESS/COMPLETED/FAILED) is identical
everywhere, but the *usage* has drifted into bugs: only bundling sets `IN_PROGRESS`;
archive's stuck-recovery resets healthy long-running transfers because it never sets it
(#151); recovery is keyed by free-text strings that drift apart (recovery handler `archive`
vs the `long_term_archive` the task writes → stalled archives never recover, #69; packaging
never sets its kind at all). Counters are inconsistent (`retry_count` /
`unpack_retry_count` / `attempt_count`; `PhysicalCopy` persists none — #90, #75).

Operator-chosen rollback (#149, "walk up the pipeline state chain") is blocked on this: it
needs every stage to be a uniform, independently addressable operation. This ADR
establishes that model so #149 is purely additive.

**Why now:** the database is disposable (re-seed is cheap, no production data), so these are
clean schema migrations with **no backfill** — the single largest risk is absent. That
window closes when stable operations begin; this is the moment to pay the schema cost.

## Decision

Adopt one uniform operation model. Six coupled choices:

1. **One polymorphic `Operation` base table (joined-table inheritance)**, the same pattern
   as `SystemLog` / `PhysicalCopy`. Subclasses carry stage-specific fields:
   `PackagingOperation`, `BundlingOperation`, `TransferOperation`, `UnpackOperation`,
   `ArchiveOperation`, `StagingOperation`. **Packaging and bundling are lifted** off their
   artifacts: `RawDataPackage` / `DataTransferPackage` keep only their lifecycle `state`
   (`PackageState`), and operation `status` moves to the `Operation` row. The transfer/
   unpack unicorn record splits into two sibling rows (this realizes ops-db#84, which this
   ADR **subsumes**).

2. **One canonical operation state machine.** Every operation walks
   `PENDING → SCHEDULED → IN_PROGRESS → COMPLETED/FAILED`. `IN_PROGRESS` is **mandatory**
   and set in **one place** — the `CCATEnhancedSQLAlchemyTask` base (which already owns the
   start hook + heartbeat) — giving the DB an honest "running" signal and structurally
   fixing the #151 class for all stages. Packaging adopts the standard
   **poll→SCHEDULED→dispatch** find-work loop (dropping inline create-and-dispatch), making
   it re-runnable from `PENDING` and closing its double-dispatch race (#70-class). One
   **`retry_count`** on the base replaces `attempt_count` / `unpack_retry_count` and gives
   #75 a single `get_retry_count()`.

3. **One `OperationKind(str, Enum)` whose values equal the frozen breadcrumb strings**
   (`raw_data_package`, `data_transfer_package`, `transfer`, `unpack`, `long_term_archive`,
   `staging`). It is the single source of truth for the *identity* axis — recovery, circuit
   breaker, `OperationFailureEvent` key (ADR-0001), and the JTI discriminator all reference
   the enum member, never a literal. Because values equal already-stored strings, this is a
   **code refactor, not a data migration**, and it makes the #69 drift class
   unrepresentable. The existing stale literal (recovery's `archive`) is reconciled to
   `OperationKind.ARCHIVE` (= `long_term_archive`) on adoption. The **routing axis**
   (`OperationType.value` + `QUEUE_OPERATIONS_BY_LOCATION_TYPE`) is a separate, frozen
   contract: queue names are built from it and workers self-discover queues at startup, so
   nothing here changes how workers run. Identity (`operation_kind`) and routing
   (`operation_type`) must stay distinct in code.

4. **Artifact/copy-anchored lineage.** An `Operation` *consumes* input `PhysicalCopy`(s)
   and *produces* output `PhysicalCopy`(s); this produce/consume contract is identical for
   every kind. Lineage is traced through the artifact/copy graph
   (`RawDataFile → RawDataPackage → DataTransferPackage`), **not** through
   operation-to-operation FKs — because the pipeline's fan-in (bundling) and fan-out
   (unpack) live between artifacts. Transfer and unpack are **siblings on the same
   `DataTransferPackage`**, not unpack-hung-off-transfer. This is the walk-up seam #149
   builds on.

5. **`OperationGroup` for batched operations.** A generic polymorphic parent that bundles N
   co-created operations and carries a status *derived* from its children (never set
   independently). `StagingJob` becomes `OperationGroup` **plus a retention claim** — its
   `active` flag holds the staged packages until released (`deletion_manager` already
   enforces this). The base is built minimal/claim-free now; its known future second
   customer is grouping the multiple results of a workflow reduction step.

6. **Deletion stays out, on shape grounds.** Deletion is a `PhysicalCopyStatus` state
   machine *on the artifact* (`PRESENT → DELETION_* → DELETED`), running in the destroy
   direction — not a unit of work with a `Status`. Forcing it into the `Operation` model
   would fabricate an op row for an artifact transition. Its missing counter (#90) is fixed
   separately on `PhysicalCopy`.

Migration is **additive and incremental**: build the tables, migrate **one stage at a
time** behind `reconcile_package_state` / `classify_transferring_step`, **staging-first**,
leaning on the ops-db#78 Alembic drift gate. New/moved JSON columns land as `jsonb`, not
`json` (ops-db#92).

## Considered options

- **Status stays on the artifact (no lift).** Rejected: leaves packaging/bundling as
  snowflakes, so "every stage is an operation" stays false for two of five and #149 can't
  address them uniformly. The act of packaging and the package artifact are genuinely
  separate concerns, and `reconcile_package_state` already derives `state` from operation
  statuses (ADR-0004, ops-db-api).
- **Separate per-stage tables sharing an `OperationMixin` (uniform shape, no base table).**
  Lower blast radius and closest to today, but uniformity is by convention not schema,
  `operation_id` keeps colliding across tables (so `OperationFailureEvent` can never become
  a real FK), and #149's "same handle" stays a code-level abstraction over heterogeneous
  tables. Rejected for a disposable-DB window where the stronger cut is affordable.
- **Operation-chained lineage** (`upstream_operation_id`, unpack 1:1 off transfer).
  Rejected: a 1:1 chain can't represent bundling fan-in / unpack fan-out, which live
  between artifacts.
- **Keep raw `operation_type` strings** (no enum). Rejected: preserves the stringly-typed
  drift that produced #69 and the packaging mismatch.
- **New clean `OperationKind` enum + translation layer** to the frozen strings. Rejected:
  the mapping boundary is a *new* drift surface that reintroduces the #69 class and forces
  a history consideration — strictly worse than values-equal-frozen-strings.
- **Merge the identity and routing axes into one enum.** Deferred, not chosen: routing
  values feed dynamically-discovered queue names, so changing them would silently spin up
  new queues and orphan in-flight messages (a live-drain hazard) and force coordinated
  worker redeploys. Not required for uniformity.
- **Drop `IN_PROGRESS`, rely only on the Redis heartbeat.** Rejected: loses the DB-visible
  running state the UI and stuck-detection need; the heartbeat stays the truth for
  liveness, the DB state is its legible projection.
- **Fold deletion into the `Operation` model.** Rejected on shape (see Decision 6).

## Consequences

- **Cross-repo, lockstep.** ops-db (models + migrations) → ops-db-api
  (`reconcile_package_state`, `classify_transferring_step`, attribution, reset endpoints —
  all read the duality directly today and must move together; ADR-0004 preserved) →
  data-transfer managers + base task → ops-db-ui. The UI is **more** coupled than first
  assumed: 8–12 components branch on operation kind and read paired fields and per-kind
  failure counts, so it does not "barely move."
- **Realizes and supersedes ops-db#84**; folds in #69, #75, the LTA counter; **generalizes
  #151** (the per-stage `IN_PROGRESS` fix). #151 remains the immediate standalone fix.
- **Breadcrumbs (ADR-0001) stay correct.** `OperationKind` values equal the frozen
  `OperationFailureEvent.operation_type` strings, so existing history stays correlatable;
  with a single global `operation_id` the loose `(operation_type, operation_id)` key *may*
  later become a real FK (a follow-up, not required here).
- **PackageState (`state`) is unchanged** — it remains the artifact lifecycle, derived from
  operation statuses; it is not absorbed into `Operation.status`.
- **Enables #149** by exposing every stage as a uniform operation with consumed/produced
  `PhysicalCopy` lineage; the rollback verb itself stays out of scope.
- **No backfill, no production downtime** because the DB is disposable — the deciding factor
  in choosing the deeper cut and in timing it before stable operations.