I/O Layer#

The I/O layer handles all Zarr and FITS interactions, isolating the math layer (cal-core) from filesystem concerns.

L0 Zarr Reading#

File: cal-io/src/raw.rs

Key functions:

  • open_store(path) — opens a Zarr store (auto-detects v2/v3)

  • list_scans(path) — discovers scan groups by naming pattern

  • load_scan(path, scan_number) — loads source data as ScanData

  • load_calibration(path, scan_number) — loads HOT/COLD data as CalibrationSnapshot

5D Data Loading#

The 5D count array [C, D, R, A, S] is read with two strategies:

  • Small arrays (< 10M elements): read entire array via zarrs, then split into per-subscan Array4 views

  • Large arrays: parallel per-subscan chunk decompression via Rayon, each subscan assembled contiguously (avoids zarrs scatter-write overhead on large datasets)

i32::MIN values (FITS fill) are converted to NaN (f64) during the read.

L1 Zarr Writing#

File: cal-io/src/calibrated.rs

write_calibration_result() creates a scan group in the output store and writes all output arrays with zstd compression. Chunking uses min(1024, dim[0]) for the channel axis and 1 for remaining axes.

Scan-level attributes store metadata (source, mode, PWV, QA metrics).

FITS Ingest#

File: cal-io/src/ingest.rs (behind fits-ingest feature)

convert_fits_to_zarr(fits_folder, zarr_path, skip_if_complete) wraps zarr-fits-core’s convert_folder() with:

  • Skip-if-complete — checks for _zarr_fits_complete.json marker

  • Default path derivation — /data/raw/session → session.zarr

  • Error bridging — converts ZarrFitsError to CalibrationError

The calibrate -f CLI flag calls this before calibration.

Schema Constants#

File: cal-schema/src/lib.rs

All L0 and L1 array names (data_5d, sobsmode, spectra, t_sys, etc.) are defined once in the cal-schema crate and re-exported by both cal-io and zarr-fits-core. This prevents the reader and writer from disagreeing on array names.