---
status: accepted
---

# Live cross-host filesystem-check contract (`diagnostics.stat_paths`)

ops-db-api #111, epic data-transfer #139. Worker side: data-transfer #141.

## Context

The DB-only diagnosis (#110) reports what the database *believes* about an
operation's files. The next tier asks the per-site hosts whether the bytes are
actually on disk *right now*. ops-db-api owns the admin trigger/poll API; the
data-transfer workers own the actual `stat` on each host (they alone are
deployed at the sites, with the mounts and credentials). The two repos are
separate codebases — **ops-db-api must not import `ccat_data_transfer`** — so
the only thing binding producer and worker is a wire contract, which this ADR
pins.

Two further constraints shaped the design:

- **Broker vs. buffering Redis.** In observatory deployments
  `ccat_ops_db_api_settings.redis_host` is `redis-ccat`, the read-only
  transaction-**buffering** Redis — **not** the Celery broker. The existing
  `dependencies.get_redis()` points at that buffering Redis. Dispatching the
  stat task there would put it on a Redis no worker consumes from.
- **No new DB table.** The correlation between a poll request and its dispatched
  tasks is short-lived operational state, not a durable record.

## Decision

**The contract** between the ops-db-api producer and the data-transfer worker is:

- **Task name** — `diagnostics.stat_paths`.
- **Args** — a single positional argument: the list of path strings to stat.
  The producer calls `send_task("diagnostics.stat_paths", args=[[p1, p2, …]])`.
- **Per-site queue routing** — one task per relevant location, routed to the
  owning `Site.short_name` queue (`cologne` / `us` / `fyst`). These are the
  **site-level static queues** that data-transfer's `disk_monitor` declares and
  the stat worker (#141) consumes from — NOT the compound
  `{short_name}_{name}_{operation}` queues that `route_task_by_location` builds
  for the transfer/monitor tasks.
- **Result shape** — the worker returns a list of per-path dicts, each
  `{path, exists, size, readable, error}` (`size`/`readable` null when absent,
  `error` a string when the stat itself failed).

**The producer** (`ccat_ops_db_api/live_check/producer.py`) is a *producer-only*
Celery app built against a **new, separate** broker connection:

- New settings `celery_broker_host` / `celery_broker_port` / `celery_broker_db`
  / `celery_broker_ssl_*`, kept **explicitly distinct** from `redis_host`. They
  point at the MAIN broker (the broker data-transfer's workers consume from) and
  reuse the existing `/etc/redis/certs/` SSL material (`redis_ca_cert` /
  `redis_certfile` / `redis_keyfile`). They default to localhost so local dev
  and CI need no broker.
- The producer's **result backend uses the same `celery_broker_db` index** as
  the broker. This index **MUST match the data-transfer worker's `REDIS_DB`**:
  the worker writes its stat result to its `REDIS_DB`, and the poll endpoint
  reads `AsyncResult` from `celery_broker_db`. A mismatch silently lands results
  in a different db and the poll stays `pending` forever (no error surfaces).
- It dispatches purely by task name via `send_task`; it registers **no** worker
  code. `get_redis()` is **not** reused for dispatch.

**Only DISK locations are filesystem-checked.** A site can own both a disk
buffer and an S3 `long_term_archive` under the same `short_name` (the production
Cologne topology). Since the worker stats with `os.stat`, an S3/tape object key
would always read as absent and falsely report a present archive as "missing".
The producer skips non-disk copies and surfaces them in the trigger response's
`skipped` list so the caller knows they were not live-checked (the #110 DB
diagnosis still covers them).

**The correlation mapping** `{correlation_id → [(task_id, host_label, queue), …]}`
is a TTL'd Redis key on the **MAIN broker** Redis (the producer's own sync
client, `producer.get_broker_redis()`), never on the buffering `redis-ccat`. The
poll endpoint reads the mapping and each task's `AsyncResult`, then reduces each
host's per-path dicts to one status: `error` if any path errored, else `missing`
if any path is absent, else `present`; an unready task is `pending`; a vanished
correlation key is `state="expired"`. A finished task with an **empty/falsy
result** is `error`, not `present` — every dispatched host has ≥1 path, so an
empty result means the payload expired/was evicted from the result backend, and
reporting `present` there would be a false all-OK on a diagnostic tool.

## Contract test and its limitation

A TRUE cross-repo round-trip (API → real data-transfer worker) is **not
feasible in ops-db-api CI**: the worker isn't installed here and can't be
imported. `tests/test_stat_paths_contract.py` is the strongest self-contained
equivalent and exercises the **real producer `send_task` path**:

- The producer dispatches over an **in-memory broker** (`memory://`). The test
  drains the per-site queue and inspects the published message, pinning the task
  **name**, the **args** (`[[paths]]`), and the **queue** routing on the wire.
  (`task_always_eager` is deliberately *not* used: Celery's `send_task` ignores
  it — `AlwaysEagerIgnored` — so the memory transport is what removes the live
  broker.)
- A local task registered under the exact name `diagnostics.stat_paths` produces
  the `{path, exists, …}` result, which is fed through the **real poll-assembly
  logic** to prove the result shape drives the per-host status.

**Limitation:** the worker body in the test is a stand-in, not data-transfer's
real implementation. The contract is pinned by *shape and routing*, not by the
worker's actual `stat` behaviour. When data-transfer #141 lands, the real
worker must register a task under this exact name, accept `args=[[paths]]`,
consume the `cologne`/`us`/`fyst` queues, and return the per-path dict shape.
Any change to name, args, routing, or result keys is a breaking change to this
contract and must update both repos together.

## Consequences

- Two Redis connections from ops-db-api: the buffering `redis-ccat`
  (`get_redis()`, unchanged) and the new MAIN broker (`producer`). Operators must
  configure `celery_broker_*` to the real broker per environment; the localhost
  default is a dev/CI convenience, not a production value.
- The live tier is best-effort and ephemeral: if the broker is unreachable the
  trigger returns 503; if the correlation key expires the poll returns
  `expired`. No durable record is kept (no new table), matching the
  operational, throwaway nature of a "is it on disk right now?" check.
- `celery` becomes a direct dependency of ops-db-api (it was previously only
  transitive).