Archive recovery: idempotent completion over at-least-once retries#

Context#

Long-term-archive (LTA) transfers are recovered by three independent mechanisms, any of which can re-run the same archive:

  • Celery broker redelivery — the app runs task_acks_late=True + task_reject_on_worker_lost=True with Redis visibility_timeout=3600 (setup_celery_app.py). A worker that dies mid-task does not ack, so the broker redelivers the message.

  • task_monitor_service heartbeat recovery — resets a task whose Redis heartbeat has gone stale (HEARTBEAT_TIMEOUT=300) to PENDING; the archive poll loop re-dispatches it.

  • The archive manager’s own stuck path (_get_stuck_transfers) — historically reset anything SCHEDULED/IN_PROGRESS past a blind 60-min DB clock.

All three are at-least-once; acks_late explicitly permits duplicates. Two live bugs sit on top of this:

  • #151: the archive task never set IN_PROGRESS, and the stuck path judged “stuck” purely on the 60-min start_time clock — so a legitimately long-running archive (large tape/S3 transfer) was reset while still running → duplicate concurrent archive of the same package to the same LTA.

  • #69: the recovery handler was keyed archive while the task registers operation_type = "long_term_archive", so heartbeat recovery never fired for archive at all.

A decisive infrastructure constraint: the production LTA queue (cologne_cologne_long_term_archive_long_term_archive_transfer) is served by a single worker container (docker-compose.production.input-a.yml), with no replicas. Redis’s redelivery sweep runs inside a live worker — so if that one container dies ungracefully or crash-loops, the broker does not redeliver. The broker is therefore not a reliable standalone backstop here.

This ADR is the archive-scoped standalone fix, landing before the uniform-operation model (ADR-0003 / #150 / #152) generalizes it.

Decision#

  1. Correctness rests on idempotent completion, not on any single recovery path. The archive task no-ops if a RawDataPackagePhysicalCopy for (raw_data_package, destination DataLocation) already exists (plus a COMPLETED early-return at task head, mirroring data_transfer_package_manager). This neutralizes duplicates from every racing recovery path — broker redelivery, heartbeat reset, and the reconciler — by construction. Implemented in-code now; the airtight DB unique constraint on (raw_data_package_id, destination_data_location_id) is deferred to #70 (cross-repo ops-db migration).

  2. The archive task sets status = IN_PROGRESS at start. This gives the DB an honest running signal and makes a healthy long-running archive heartbeat-visible, closing the ADR-0003 “never set IN_PROGRESS” anti-pattern for the archive stage. It is not the duplicate guard (the broker/heartbeat use the Redis task status, not the DB row); the idempotency check in Decision 1 is.

  3. #69 alignment (interim). The recovery sites key on the literal long_term_archive the task actually registers (task_monitor_service handler dict + _get_operation_details, and the duplicate _get_operation_details on the base task). This makes task_monitor heartbeat recovery the primary dead-worker recovery for archive. It re-commits the free-text-string drift ADR-0003 flags, accepted as a deliberate interim — #152/#153 replaces the literal with OperationKind.

  4. Keep the manager-side net, but heartbeat-gate it. _get_stuck_transfers is converted from a blind time threshold to a heartbeat-gated reconciler: it resets SCHEDULED/IN_PROGRESS → PENDING only when no task is alive or registered (is_operation_alive on TaskStateManager, reading tasks_for_operation:long_term_archive:{id} → task heartbeats) and start_time is older than a floor ≥ visibility_timeout (3600s). It is the backstop for the cases the broker and heartbeat structurally miss: a worker dying before it registers a heartbeat (no task: hash for task_monitor to see), and DB/Redis divergence (hash TTL expiry, Redis flush). The floor keeps it from ever racing the broker’s own redelivery window. The attempt_count < 3 cap is retained, and the reconciler logs a WARNING on every reset so dead-worker / never-registered cases surface to operators.

Considered options#

  • Delete the manager net; rely solely on broker + heartbeat. Rejected: the single-worker LTA queue makes broker redelivery unreliable, and task_monitor is blind to never-registered tasks — leaving silently-stuck archives, which for a preservation pipeline is worse than a duplicate.

  • COMPLETED-only early-return as the duplicate guard. Rejected: it does not stop two concurrently-IN_PROGRESS tasks (e.g. heartbeat reset re-dispatch racing a still-running partitioned worker). The physical-copy-existence check does.

  • Heartbeat-gate the existing path at the old 60-min threshold. Rejected: a floor below visibility_timeout lets the reconciler race the broker’s redelivery; the floor must sit above it.

  • Add the DB unique constraint now. Deferred to #70 — it is a cross-repo ops-db migration. The in-code guard closes the practical case; the constraint makes it airtight (and closes the residual TOCTOU window on the in-code check).

Consequences#

  • Duplicate archive is prevented by construction. No combination of racing recovery paths can produce a second physical copy, present or future.

  • task_monitor (heartbeat) is the primary recovery; the reconciler is an explicit, WARNING-logged backstop for the windows the broker and heartbeat miss.

  • A residual TOCTOU window on the in-code copy check remains until #70’s unique constraint lands.

  • Superseded by #152. ADR-0003’s uniform model replaces the literal op-kind with OperationKind, sets IN_PROGRESS in one place for all stages, and unifies stuck-detection on the heartbeat. This ADR is the archive-scoped interim those build on — but the idempotent-completion invariant (Decision 1) is durable and must be preserved by #152.

  • Stale circuit_breaker:long_term_archive:* keys may sit OPEN after #69 until the 3600s auto-reset; a deploy note, no code.

  • No test infrastructure change: recovery/idempotency is exercised with SQLite + MagicMock Redis (the suite has no live broker; Celery tests are CI-excluded).