Filesystem boundary errors: wrap for diagnosis, keep non-retryable#

Context#

Several pipeline filesystem touch-points call os/shutil directly with no guard, so a missing source raises a bare FileNotFoundError whose str() is just [Errno 2] No such file or directory: '<path>' (e.g. transfer_manager.py:585 cp getsize; archive_manager.py:920/926 & 1068/1073 archive upload source; raw_data_package_manager.py:216/303 shutil.copy2). should_retry (setup_celery_app.py) hard-codes FileNotFoundError as non-retryable, so these go straight to permanent failure with no operation context and no explanation of why the file is gone.

Decision#

Introduce a typed SourceMissingError(StorageError) and a DB-agnostic helper (ensure_readable(path, *, side, step, host, diagnose=None)) called at the boundary points. The helper fills the Tier-2 error_context fields (step, side, path, host) and runs a caller-supplied diagnose closure that queries PhysicalCopy for the (package, location) and produces the diagnosis:

  • status DELETED → “source removed by deletion (task_id=X) at deleted_at”;

  • status PRESENT → “DB says PRESENT but file absent — possible transient mount glitch or DB/disk inconsistency”;

  • no record → “no PhysicalCopy at this location — package may never have been created here.”

SourceMissingError is raised with is_retryable=False — preserving today’s behavior exactly. B is purely about breadcrumbs, not retry policy.

Considered options#

  • (a) Keep non-retryable for all source-missing (chosen). Zero change to retry behavior; the diagnosis distinguishes truly-gone (DELETED) from possibly-transient (PRESENT-but-missing), and a manual frontend reset covers the transient case.

  • (b) Diagnosis-driven retry: PRESENT-but-missing gets bounded auto-retry for transient mount blips. Rejected for now — adds churn on genuinely-gone files and changes failure behavior; revisit only if staging shows real transient-mount churn.

Consequences#

  • SourceMissingError must set is_retryable=False explicitly, because StorageError defaults to retryable (unlike the bare FileNotFoundError it replaces).

  • The diagnoser is expected to confirm that many failures are the DELETED case — i.e. a deletion-vs-pending-work ordering race. That is a genuine pipeline bug that B only surfaces; the fix is tracked separately, not in this work.

  • The diagnoser starts with the source-missing case only (the actual pain), structured so further per-error diagnosers can be added later.