Transfer Model#
ops-db tracks the data-movement artifacts but does not perform the movement.
This page is the as-built reference for the artifact tables — the packages and
the archive-transfer record — and for how they link to the pipeline operations
that act on them. The operations themselves (the Operation family, its states,
groups, and lineage) are described in Operation Model; routing
configuration is in Location Model.
Routing configuration#
DataTransferRoute#
DataTransferRoute defines how data should flow
between sites, at site level with optional location-level overrides. This
decouples transfer logic from hardcoded paths. The route_type
(RouteType) is DIRECT (skip destination
buffer), RELAY (route through an intermediate site), or CUSTOM
(location-to-location override).
For complete attribute details, see
DataTransferRoute.
Artifact tables#
RawDataPackage#
RawDataPackage is the tar-archive bundle of
related RawDataFile objects grouped per
ExecutedObsUnit and InstrumentModule. It is produced by a
PackagingOperation and is the unit that is
bundled for transfer, archived, and staged.
It retains two residual lifecycle columns that are not an operation’s own state:
state(PackageState) — coarse UI status (WAITING/TRANSFERRING/ARCHIVED/FAILED).analyze_status(Status) — readiness for scientific analysis.
For complete attribute details, see
RawDataPackage.
DataTransferPackage#
DataTransferPackage bundles multiple
RawDataPackage objects into one network-transfer
unit (optimal sizes are in the 10–50 TB range for long-distance links),
reducing the overhead of many small transfers.
It is produced by a BundlingOperation, then moved
by one or more TransferOperation rows and
extracted/verified by sibling UnpackOperation
rows (one per destination). It carries no state column: its lifecycle is
its bundling/transfer/unpack operations, not a status field on the artifact.
For complete attribute details, see
DataTransferPackage.
LongTermArchiveTransfer#
LongTermArchiveTransfer records the copy of a
RawDataPackage from a buffer to permanent
archive storage within a single site (between a BUFFER and a
LONG_TERM_ARCHIVE DataLocation), as opposed to
the cross-site move modelled by ArchiveOperation
and TransferOperation.
It retains two residual columns, last_attempt_time and error_message. Its
detailed log entries are LongTermArchiveTransferLog
rows.
For complete attribute details, see
LongTermArchiveTransfer.
StagingJob#
StagingJob makes archived data available for
scientific processing: it downloads from the long-term archive, unpacks, and
records file access. Multiple packages can be staged together. Staging is
on-demand (a scientist requests data), unlike archiving, which is automatic. In
the uniform model its successor is StagingOperation
grouped under a StagingOperationGroup; the legacy
StagingJob is retained during the additive transition.
For complete attribute details, see
StagingJob.
How artifacts link to operations#
Artifacts are nouns; operations are verbs. Each operation references the artifact it acts on by foreign key, and lineage between artifacts is traced through the copy graph rather than through the artifacts directly:
Artifact |
Produced by |
Consumed / further acted on by |
|---|---|---|
Operations consume and produce PhysicalCopy rows
(one per artifact instance at a DataLocation)
through the copy-anchored lineage tables. For the full operation structure,
states, groups, and the no-op→op-FK lineage rule, see Operation Model.
Integration with the data-transfer package#
ops-db only stores these records; the data-transfer package reads them to decide what to move, performs the transfers (BBCP, S3, etc.), and updates operation status and physical-copy records as work completes. The why of the operation model and the state-transition behaviour live there, not here — see the data-transfer ADR-0003 — Uniform pipeline operation model.