# Operation Model

```{eval-rst}
.. verified:: 2026-06-24
   :reviewer: Christof Buchbender
```

ops-db owns the *structure* of the uniform pipeline operation model: the tables,
the polymorphic inheritance, and the enum values. The *behaviour* — how an
operation moves between states, how it is dispatched, retried, and recovered —
lives in the data-transfer package. This page is the as-built reference for the
structure; for the rationale see the data-transfer
[ADR-0003 — Uniform pipeline operation model](https://ccatobs.github.io/data-center-documentation/data-transfer/docs/adr/0003-uniform-pipeline-operation-model.html),
and for how identity and routing are kept on separate axes see the data-transfer
[Operation identity vs routing](https://ccatobs.github.io/data-center-documentation/data-transfer/docs/source/operation_identity_vs_routing.html)
page.

The artifact tables that operations act on
({py:class}`~ccat_ops_db.models.RawDataPackage`,
{py:class}`~ccat_ops_db.models.DataTransferPackage`,
{py:class}`~ccat_ops_db.models.LongTermArchiveTransfer`) are described in
{doc}`transfer_model`.

## Operation

{py:class}`~ccat_ops_db.models.Operation` is the polymorphic base for every
pipeline operation, using joined-table inheritance (the same pattern as
{py:class}`~ccat_ops_db.models.SystemLog` and
{py:class}`~ccat_ops_db.models.PhysicalCopy`). The base table `operation` holds
the fields shared by every kind; each subclass adds its own table with the
stage-specific columns, joined on a shared `id`.

The discriminator column is `operation_kind`, holding an
{py:class}`~ccat_ops_db.models.OperationKind` value. `Operation` has no base
`polymorphic_identity`: every row is one of the six concrete kinds below
(mirroring {py:class}`~ccat_ops_db.models.DataLocation`'s `None` base identity).

Uniform base columns:

| Column | Type | Meaning |
|---|---|---|
| `id` | int (PK) | Operation row identity. |
| `operation_kind` | `OperationKind` | Discriminator / identity axis (frozen string value). |
| `status` | `Status` | Canonical operation state (see the Status section). |
| `retry_count` | int | Attempt counter. |
| `failure_error_message` | text | Latest failure message (denormalized). |
| `error_context` | JSONB | Latest structured failure breadcrumb. |
| `start_time` / `end_time` | datetime | Execution window (nullable). |
| `operation_group_id` | int (FK) | Optional membership in an {py:class}`~ccat_ops_db.models.OperationGroup`. |
| `consumed_copies` / `produced_copies` | relationship | Copy-anchored lineage (see the Copy-anchored lineage section). |

### Subclasses

The six concrete kinds, each a joined-table subclass adding stage-specific
foreign keys:

| Subclass | `polymorphic_identity` (`OperationKind`) | Stage-specific columns |
|---|---|---|
| {py:class}`~ccat_ops_db.models.PackagingOperation` | `PACKAGING` | `raw_data_package_id` |
| {py:class}`~ccat_ops_db.models.BundlingOperation` | `BUNDLING` | `data_transfer_package_id` |
| {py:class}`~ccat_ops_db.models.TransferOperation` | `TRANSFER` | `data_transfer_package_id`, `origin_location_id`, `destination_location_id`, `transfer_method` |
| {py:class}`~ccat_ops_db.models.UnpackOperation` | `UNPACK` | `data_transfer_package_id`, `destination_location_id` |
| {py:class}`~ccat_ops_db.models.ArchiveOperation` | `ARCHIVE` | `raw_data_package_id`, `origin_location_id`, `destination_location_id` |
| {py:class}`~ccat_ops_db.models.StagingOperation` | `STAGING` | `raw_data_package_id`, `origin_location_id`, `destination_location_id` |

`TransferOperation` and `UnpackOperation` are siblings on the same
{py:class}`~ccat_ops_db.models.DataTransferPackage`: the move and the
extract/verify are independent rows. `UnpackOperation` is anchored per
`(data_transfer_package_id, destination_location_id)`, so a package transferred
to several destinations is unpacked once per destination, mirroring its sibling.

## OperationKind

{py:class}`~ccat_ops_db.models.OperationKind` is a `str`-subclassing enum and the
single source of truth for the identity axis. The six values are byte-identical
to the frozen breadcrumb strings stored in
{py:class}`~ccat_ops_db.models.OperationFailureEvent`'s `operation_type` and used
by recovery, the circuit breaker, and routing.

| Name | Value |
|---|---|
| `PACKAGING` | `raw_data_package` |
| `BUNDLING` | `data_transfer_package` |
| `TRANSFER` | `transfer` |
| `UNPACK` | `unpack` |
| `ARCHIVE` | `long_term_archive` |
| `STAGING` | `staging` |

Two members carry a deliberate name/value **asymmetry**: `PACKAGING ==
"raw_data_package"` and `ARCHIVE == "long_term_archive"`. The names follow the
pipeline stage; the values follow the pre-existing frozen breadcrumb strings.
The other four names match their values.

### `.value` wire safety

Always store and compare `OperationKind` via its `.value`, never via
`str(member)` or an f-string. `OperationKind` subclasses `str`, but under
Python 3.12 `str(OperationKind.PACKAGING)` and `f"{OperationKind.PACKAGING}"`
yield the member *name* (`"OperationKind.PACKAGING"`), not the value
(`"raw_data_package"`). Because these strings key Redis entries and are matched
against the frozen breadcrumb strings, the name form corrupts Redis keys and
breaks the lookup. The discriminator column persists `.value`
(`values_callable=lambda x: [e.value for e in x]`), so the database is already
safe; the hazard is in application code that serializes a member by hand.

## Operation groups

{py:class}`~ccat_ops_db.models.OperationGroup` bundles N co-created operations.
Its `status` is **derived**, exposed as a read-only Python property — never a
mapped column and never set independently — so the derivation rule is the single
source of truth and the group cannot drift from its children:

- any child `FAILED` → `FAILED`
- all children `COMPLETED` → `COMPLETED`
- empty, or all `PENDING`/`SCHEDULED` → `PENDING`
- otherwise (work in flight) → `IN_PROGRESS`

{py:class}`~ccat_ops_db.models.StagingOperationGroup` is the staging
specialization (joined-table subclass of `OperationGroup`). It adds the boolean
`active` retention claim, which holds the staged packages' copies until released;
the status is still derived on the base.

## Copy-anchored lineage

Lineage is **copy-anchored**. Every operation consumes input
{py:class}`~ccat_ops_db.models.PhysicalCopy` rows and produces output
`PhysicalCopy` rows, via the two association tables
`operation_consumes_physical_copy` and `operation_produces_physical_copy`
(exposed as the `consumed_copies` / `produced_copies` relationships). There are
**no operation-to-operation foreign keys**: provenance is traced through the
artifact/copy graph, not through op→op links. The consume/produce split is two
tables rather than one with a role column, so each direction is independently
queryable from either side, and the pipeline's fan-in (bundling) and fan-out
(unpack) live between artifacts rather than on the operations.

## Status

{py:class}`~ccat_ops_db.models.Status` is the canonical operation state machine,
shared uniformly across the pipeline. This reference lists the states only; the
allowed transitions and the recovery behaviour live in data-transfer (see
[ADR-0003](https://ccatobs.github.io/data-center-documentation/data-transfer/docs/adr/0003-uniform-pipeline-operation-model.html)).

| Value | Meaning |
|---|---|
| `PENDING` | Waiting to be scheduled; the not-yet-started default. |
| `SCHEDULED` | Dispatched for execution but not yet picked up. |
| `IN_PROGRESS` | A worker is actively executing the operation. |
| `COMPLETED` | Finished successfully. |
| `FAILED` | Execution failed. |

## OperationFailureEvent

{py:class}`~ccat_ops_db.models.OperationFailureEvent` is the append-only failure
trail — one row per failure occurrence (retryable or permanent), never erased by
a reset or retry. It is the durable history behind the denormalized per-row
`error_context` cache on `Operation`.

Being generic across operation kinds, it carries **no foreign key** to an
operation row (it cannot reference one of several subclass tables). Instead it
keys on the `(operation_type, operation_id)` pair the system already correlates
on, where `operation_type` is the `OperationKind` value. Its `error_context`
column (JSONB) holds the structured failure breadcrumb for that occurrence.

## Artifact residue (post-#95)

After the operation model absorbed per-stage lifecycle, the artifact tables keep
only the residual columns that are not an operation's own state:

| Artifact | Residual lifecycle columns | Note |
|---|---|---|
| {py:class}`~ccat_ops_db.models.RawDataPackage` | `state`, `analyze_status` | Retains both. |
| {py:class}`~ccat_ops_db.models.DataTransferPackage` | *(none)* | Has **no** `state`; its lifecycle is its {py:class}`~ccat_ops_db.models.BundlingOperation`. |
| {py:class}`~ccat_ops_db.models.LongTermArchiveTransfer` | `last_attempt_time`, `error_message` | Retains both. |

See {doc}`transfer_model` for the full artifact tables and how they link to
operations.