Operation Model#

✓

Documentation Verified Last checked: 2026-06-24 Reviewer: Christof Buchbender

ops-db owns the structure of the uniform pipeline operation model: the tables, the polymorphic inheritance, and the enum values. The behaviour — how an operation moves between states, how it is dispatched, retried, and recovered — lives in the data-transfer package. This page is the as-built reference for the structure; for the rationale see the data-transfer ADR-0003 — Uniform pipeline operation model, and for how identity and routing are kept on separate axes see the data-transfer Operation identity vs routing page.

The artifact tables that operations act on (RawDataPackage, DataTransferPackage, LongTermArchiveTransfer) are described in Transfer Model.

Operation#

Operation is the polymorphic base for every pipeline operation, using joined-table inheritance (the same pattern as SystemLog and PhysicalCopy). The base table operation holds the fields shared by every kind; each subclass adds its own table with the stage-specific columns, joined on a shared id.

The discriminator column is operation_kind, holding an OperationKind value. Operation has no base polymorphic_identity: every row is one of the six concrete kinds below (mirroring DataLocation’s None base identity).

Uniform base columns:

Column	Type	Meaning
`id`	int (PK)	Operation row identity.
`operation_kind`	`OperationKind`	Discriminator / identity axis (frozen string value).
`status`	`Status`	Canonical operation state (see the Status section).
`retry_count`	int	Attempt counter.
`failure_error_message`	text	Latest failure message (denormalized).
`error_context`	JSONB	Latest structured failure breadcrumb.
`start_time` / `end_time`	datetime	Execution window (nullable).
`operation_group_id`	int (FK)	Optional membership in an `OperationGroup`.
`consumed_copies` / `produced_copies`	relationship	Copy-anchored lineage (see the Copy-anchored lineage section).

Subclasses#

The six concrete kinds, each a joined-table subclass adding stage-specific foreign keys:

Subclass	`polymorphic_identity` (`OperationKind`)	Stage-specific columns
`PackagingOperation`	`PACKAGING`	`raw_data_package_id`
`BundlingOperation`	`BUNDLING`	`data_transfer_package_id`
`TransferOperation`	`TRANSFER`	`data_transfer_package_id`, `origin_location_id`, `destination_location_id`, `transfer_method`
`UnpackOperation`	`UNPACK`	`data_transfer_package_id`, `destination_location_id`
`ArchiveOperation`	`ARCHIVE`	`raw_data_package_id`, `origin_location_id`, `destination_location_id`
`StagingOperation`	`STAGING`	`raw_data_package_id`, `origin_location_id`, `destination_location_id`

TransferOperation and UnpackOperation are siblings on the same DataTransferPackage: the move and the extract/verify are independent rows. UnpackOperation is anchored per (data_transfer_package_id, destination_location_id), so a package transferred to several destinations is unpacked once per destination, mirroring its sibling.

OperationKind#

OperationKind is a str-subclassing enum and the single source of truth for the identity axis. The six values are byte-identical to the frozen breadcrumb strings stored in OperationFailureEvent’s operation_type and used by recovery, the circuit breaker, and routing.

Name	Value
`PACKAGING`	`raw_data_package`
`BUNDLING`	`data_transfer_package`
`TRANSFER`	`transfer`
`UNPACK`	`unpack`
`ARCHIVE`	`long_term_archive`
`STAGING`	`staging`

Two members carry a deliberate name/value asymmetry: PACKAGING == "raw_data_package" and ARCHIVE == "long_term_archive". The names follow the pipeline stage; the values follow the pre-existing frozen breadcrumb strings. The other four names match their values.

`.value` wire safety#

Always store and compare OperationKind via its .value, never via str(member) or an f-string. OperationKind subclasses str, but under Python 3.12 str(OperationKind.PACKAGING) and f"{OperationKind.PACKAGING}" yield the member name ("OperationKind.PACKAGING"), not the value ("raw_data_package"). Because these strings key Redis entries and are matched against the frozen breadcrumb strings, the name form corrupts Redis keys and breaks the lookup. The discriminator column persists .value (values_callable=lambda x: [e.value for e in x]), so the database is already safe; the hazard is in application code that serializes a member by hand.

Operation groups#

OperationGroup bundles N co-created operations. Its status is derived, exposed as a read-only Python property — never a mapped column and never set independently — so the derivation rule is the single source of truth and the group cannot drift from its children:

any child FAILED → FAILED
all children COMPLETED → COMPLETED
empty, or all PENDING/SCHEDULED → PENDING
otherwise (work in flight) → IN_PROGRESS

StagingOperationGroup is the staging specialization (joined-table subclass of OperationGroup). It adds the boolean active retention claim, which holds the staged packages’ copies until released; the status is still derived on the base.

Copy-anchored lineage#

Lineage is copy-anchored. Every operation consumes input PhysicalCopy rows and produces output PhysicalCopy rows, via the two association tables operation_consumes_physical_copy and operation_produces_physical_copy (exposed as the consumed_copies / produced_copies relationships). There are no operation-to-operation foreign keys: provenance is traced through the artifact/copy graph, not through op→op links. The consume/produce split is two tables rather than one with a role column, so each direction is independently queryable from either side, and the pipeline’s fan-in (bundling) and fan-out (unpack) live between artifacts rather than on the operations.

Status#

Status is the canonical operation state machine, shared uniformly across the pipeline. This reference lists the states only; the allowed transitions and the recovery behaviour live in data-transfer (see ADR-0003).

Value	Meaning
`PENDING`	Waiting to be scheduled; the not-yet-started default.
`SCHEDULED`	Dispatched for execution but not yet picked up.
`IN_PROGRESS`	A worker is actively executing the operation.
`COMPLETED`	Finished successfully.
`FAILED`	Execution failed.

OperationFailureEvent#

OperationFailureEvent is the append-only failure trail — one row per failure occurrence (retryable or permanent), never erased by a reset or retry. It is the durable history behind the denormalized per-row error_context cache on Operation.

Being generic across operation kinds, it carries no foreign key to an operation row (it cannot reference one of several subclass tables). Instead it keys on the (operation_type, operation_id) pair the system already correlates on, where operation_type is the OperationKind value. Its error_context column (JSONB) holds the structured failure breadcrumb for that occurrence.

Artifact residue (post-#95)#

After the operation model absorbed per-stage lifecycle, the artifact tables keep only the residual columns that are not an operation’s own state:

Artifact	Residual lifecycle columns	Note
`RawDataPackage`	`state`, `analyze_status`	Retains both.
`DataTransferPackage`	(none)	Has no `state`; its lifecycle is its `BundlingOperation`.
`LongTermArchiveTransfer`	`last_attempt_time`, `error_message`	Retains both.

See Transfer Model for the full artifact tables and how they link to operations.