---
status: accepted
---

# Archive recovery: idempotent completion over at-least-once retries

## Context

Long-term-archive (LTA) transfers are recovered by **three** independent mechanisms, any
of which can re-run the same archive:

- **Celery broker redelivery** — the app runs `task_acks_late=True` +
  `task_reject_on_worker_lost=True` with Redis `visibility_timeout=3600`
  (`setup_celery_app.py`). A worker that dies mid-task does not ack, so the broker
  redelivers the message.
- **`task_monitor_service` heartbeat recovery** — resets a task whose Redis heartbeat has
  gone stale (`HEARTBEAT_TIMEOUT=300`) to `PENDING`; the archive poll loop re-dispatches it.
- **The archive manager's own stuck path** (`_get_stuck_transfers`) — historically reset
  anything `SCHEDULED`/`IN_PROGRESS` past a blind 60-min DB clock.

All three are **at-least-once**; `acks_late` explicitly *permits* duplicates. Two live bugs
sit on top of this:

- **#151:** the archive task **never set `IN_PROGRESS`**, and the stuck path judged
  "stuck" purely on the 60-min `start_time` clock — so a legitimately long-running archive
  (large tape/S3 transfer) was reset *while still running* → duplicate concurrent archive of
  the same package to the same LTA.
- **#69:** the recovery handler was keyed `archive` while the task registers
  `operation_type = "long_term_archive"`, so heartbeat recovery never fired for archive at
  all.

A decisive infrastructure constraint: the production LTA queue
(`cologne_cologne_long_term_archive_long_term_archive_transfer`) is served by a **single**
worker container (`docker-compose.production.input-a.yml`), with no replicas. Redis's
redelivery sweep runs *inside a live worker* — so if that one container dies ungracefully or
crash-loops, **the broker does not redeliver**. The broker is therefore *not* a reliable
standalone backstop here.

This ADR is the **archive-scoped standalone fix**, landing before the uniform-operation
model (ADR-0003 / #150 / #152) generalizes it.

## Decision

1. **Correctness rests on idempotent completion, not on any single recovery path.** The
   archive task no-ops if a `RawDataPackagePhysicalCopy` for
   `(raw_data_package, destination DataLocation)` already exists (plus a `COMPLETED`
   early-return at task head, mirroring `data_transfer_package_manager`). This neutralizes
   duplicates from *every* racing recovery path — broker redelivery, heartbeat reset, and
   the reconciler — by construction. Implemented in-code now; the airtight DB **unique
   constraint** on `(raw_data_package_id, destination_data_location_id)` is deferred to #70
   (cross-repo ops-db migration).

2. **The archive task sets `status = IN_PROGRESS` at start.** This gives the DB an honest
   running signal and makes a healthy long-running archive heartbeat-visible, closing the
   ADR-0003 "never set `IN_PROGRESS`" anti-pattern for the archive stage. It is *not* the
   duplicate guard (the broker/heartbeat use the Redis task status, not the DB row); the
   idempotency check in Decision 1 is.

3. **#69 alignment (interim).** The recovery sites key on the literal `long_term_archive`
   the task actually registers (`task_monitor_service` handler dict + `_get_operation_details`,
   and the duplicate `_get_operation_details` on the base task). This makes `task_monitor`
   heartbeat recovery the **primary** dead-worker recovery for archive. It re-commits the
   free-text-string drift ADR-0003 flags, accepted as a deliberate interim — #152/#153
   replaces the literal with `OperationKind`.

4. **Keep the manager-side net, but heartbeat-gate it.** `_get_stuck_transfers` is converted
   from a blind time threshold to a **heartbeat-gated reconciler**: it resets
   `SCHEDULED`/`IN_PROGRESS` → `PENDING` only when no task is alive or registered
   (`is_operation_alive` on `TaskStateManager`, reading
   `tasks_for_operation:long_term_archive:{id}` → task heartbeats) **and** `start_time` is
   older than a floor `≥ visibility_timeout` (3600s). It is the **backstop** for the cases
   the broker and heartbeat structurally miss: a worker dying *before* it registers a
   heartbeat (no `task:` hash for `task_monitor` to see), and DB/Redis divergence (hash TTL
   expiry, Redis flush). The floor keeps it from ever racing the broker's own redelivery
   window. The `attempt_count < 3` cap is retained, and the reconciler logs a WARNING on
   every reset so dead-worker / never-registered cases surface to operators.

## Considered options

- **Delete the manager net; rely solely on broker + heartbeat.** Rejected: the single-worker
  LTA queue makes broker redelivery unreliable, and `task_monitor` is blind to
  never-registered tasks — leaving silently-stuck archives, which for a preservation pipeline
  is worse than a duplicate.
- **`COMPLETED`-only early-return as the duplicate guard.** Rejected: it does not stop two
  concurrently-`IN_PROGRESS` tasks (e.g. heartbeat reset re-dispatch racing a still-running
  partitioned worker). The physical-copy-existence check does.
- **Heartbeat-gate the existing path at the old 60-min threshold.** Rejected: a floor below
  `visibility_timeout` lets the reconciler race the broker's redelivery; the floor must sit
  above it.
- **Add the DB unique constraint now.** Deferred to #70 — it is a cross-repo ops-db
  migration. The in-code guard closes the practical case; the constraint makes it airtight
  (and closes the residual TOCTOU window on the in-code check).

## Consequences

- **Duplicate archive is prevented by construction.** No combination of racing recovery
  paths can produce a second physical copy, present or future.
- `task_monitor` (heartbeat) is the primary recovery; the reconciler is an explicit,
  WARNING-logged backstop for the windows the broker and heartbeat miss.
- A residual TOCTOU window on the in-code copy check remains until #70's unique constraint
  lands.
- **Superseded by #152.** ADR-0003's uniform model replaces the literal op-kind with
  `OperationKind`, sets `IN_PROGRESS` in one place for all stages, and unifies stuck-detection
  on the heartbeat. This ADR is the archive-scoped interim those build on — but the
  **idempotent-completion invariant (Decision 1) is durable and must be preserved by #152.**
- Stale `circuit_breaker:long_term_archive:*` keys may sit OPEN after #69 until the 3600s
  auto-reset; a deploy note, no code.
- No test infrastructure change: recovery/idempotency is exercised with SQLite +
  `MagicMock` Redis (the suite has no live broker; Celery tests are CI-excluded).