---
status: accepted
---

# Filesystem boundary errors: wrap for diagnosis, keep non-retryable

## Context

Several pipeline filesystem touch-points call `os`/`shutil` directly with no guard, so a
missing source raises a bare `FileNotFoundError` whose `str()` is just
`[Errno 2] No such file or directory: '<path>'` (e.g. `transfer_manager.py:585` cp
`getsize`; `archive_manager.py:920/926` & `1068/1073` archive upload source;
`raw_data_package_manager.py:216/303` `shutil.copy2`). `should_retry`
(`setup_celery_app.py`) hard-codes `FileNotFoundError` as non-retryable, so these go
straight to permanent failure with no operation context and no explanation of *why* the
file is gone.

## Decision

Introduce a typed `SourceMissingError(StorageError)` and a DB-agnostic helper
(`ensure_readable(path, *, side, step, host, diagnose=None)`) called at the boundary
points. The helper fills the Tier-2 `error_context` fields (`step`, `side`, `path`,
`host`) and runs a caller-supplied `diagnose` closure that queries `PhysicalCopy` for the
`(package, location)` and produces the `diagnosis`:

- status `DELETED` → "source removed by deletion (task_id=X) at `deleted_at`";
- status `PRESENT` → "DB says PRESENT but file absent — possible transient mount glitch or
  DB/disk inconsistency";
- no record → "no PhysicalCopy at this location — package may never have been created
  here."

`SourceMissingError` is raised with **`is_retryable=False`** — preserving today's
behavior exactly. B is purely about breadcrumbs, not retry policy.

## Considered options

- **(a) Keep non-retryable for all source-missing (chosen).** Zero change to retry
  behavior; the diagnosis distinguishes truly-gone (`DELETED`) from possibly-transient
  (`PRESENT`-but-missing), and a manual frontend reset covers the transient case.
- **(b) Diagnosis-driven retry:** `PRESENT`-but-missing gets bounded auto-retry for
  transient mount blips. Rejected for now — adds churn on genuinely-gone files and changes
  failure behavior; revisit only if staging shows real transient-mount churn.

## Consequences

- `SourceMissingError` must set `is_retryable=False` explicitly, because `StorageError`
  defaults to retryable (unlike the bare `FileNotFoundError` it replaces).
- The diagnoser is expected to confirm that many failures are the `DELETED` case — i.e. a
  **deletion-vs-pending-work ordering race**. That is a genuine pipeline bug that B only
  *surfaces*; the fix is tracked separately, not in this work.
- The diagnoser starts with the source-missing case only (the actual pain), structured so
  further per-error diagnosers can be added later.