Crate Map

The workspace is organised into seven crates with clear dependency boundaries. The key design principle is that cal-core has no I/O dependencies — it operates purely on in-memory arrays.

Dependency Graph

        graph BT
  schema[cal-schema<br/>L0/L1 constants]
  core[cal-core<br/>pure math]
  io[cal-io<br/>Zarr I/O + pipeline]
  cli[cal-cli<br/>calibrate binary]
  py[cal-py<br/>Python bindings]
  zfc[zarr-fits-core<br/>FITS→Zarr library]
  zfcli[zarr-fits-cli<br/>zarr-fits binary]

  core --> schema
  io --> core
  io --> schema
  io -.->|fits-ingest| zfc
  cli --> io
  cli --> core
  cli -.->|fits-ingest| zfc
  py --> io
  py --> core
  zfc --> schema
  zfcli --> zfc

  style schema fill:#e8eaf6
  style core fill:#e3f2fd
  style io fill:#e8f5e9
  style zfc fill:#fff3e0
    

Dashed arrows indicate optional dependencies gated by the fits-ingest feature flag.

Crate Details

cal-schema

Zero-dependency crate containing string constants for all L0 and L1 array names, group names, and schema versions. Both the writer (zarr-fits-core) and reader (cal-io) depend on this single source of truth, preventing schema drift.

Key file: crates/cal-schema/src/lib.rs

cal-core

Pure calibration math with no filesystem access. Dependencies are limited to ndarray, rayon, serde, nalgebra, and levenberg-marquardt.

Modules:

• math/ — Rayleigh-Jeans, gamma, temperature, transmission, dump averaging; includes airmass_from_elevation() in mod.rs and zenith_tau() in transmission.rs

• atm/ — AtmosphericModel trait + TableAtm interpolation

• pwv/ — PWV fitting (grid, LM, external); frequency-adaptive sigma weighting

• modes/ — observation mode preparation via a single prepare.rs (contains prepare() dispatcher for TP, OTF, OTF-chopped)

• calibrate.rs — vectorized calibrate_full() — the only calibration function

• atmosphere.rs — atmosphere determination orchestration (full, per-pixel, per-pair)

• scan/ — core data types, split into submodules:

  • scan_data.rs — ScanData

  • cal_snapshot.rs — CalibrationSnapshot (with find_hot_cold() method)

  • cal_load.rs — CalibrationLoad, CalibrationLoadFull (constructors take &CalibrationParams)

  • cal_result.rs — CalibrationResult, effective_instmode()

• config.rs — CalibrationConfig, CalibrationParams, ObsMode, PhysicsConstants (default: KalibrateCompat)

• gain.rs — gain interpolation (levels 0, 2, 3)

• flags.rs — u16 bitmask constants and utilities

• qa.rs — quality metrics

cal-io

Zarr I/O via the zarrs crate (sync API only, no async runtime) and pipeline orchestration.

Modules:

• raw.rs — L0 Zarr reader (5D data, metadata, coordinates)

• calibrated.rs — L1 Zarr writer (with provenance metadata)

• resolve.rs — bridges raw L0 into CalibrationLoad(Full)

• pipeline/ — store-level orchestration, split into submodules:

  • mod.rs — calibrate_store() entry point, prefetch thread

  • scan.rs — process_prefetched_scan(), ProvenanceContext

  • linked_pwv.rs — multi-frontend linked PWV pooling

  • dbs.rs — DBS (dual-beam-switch) pair merging

  • wrappers.rs — convenience wrappers: calibrate_single_scan(), find_next_cal_scan() (public), common_pwv_source()

• provenance.rs — L1 provenance metadata (L2 to L0 traceability)

• skydiff.rs — SKYDIFF diagnostic history

• atm_table.rs — text ATM table parser

• atm_table_bin.rs — binary ATM table I/O

• schema.rs — L1 schema helpers

• ingest.rs — FITS ingest (behind fits-ingest feature)

cal-cli

Thin binary wrapping cal-io’s pipeline with clap argument parsing, indicatif progress bar, and manifest mode. Contains zero calibration logic.

Binary name: calibrate

cal-py

Thin PyO3 surface exposing calibrate_store(), calibrate_scan_py(), list_scans(), version(), and optionally convert_fits_folder() to Python. Contains zero logic — all work is dispatched to cal-core and cal-io. Uses the vectorized path (calibrate_full) for all calibration, same as the CLI.

zarr-fits-core

FITS→Zarr v3 conversion library. Reads individual FITS files via cfitsio, groups by scan, writes Zarr v3 stores with streaming per-subscan memory.

zarr-fits-cli

Standalone zarr-fits binary for FITS conversion without calibration.

Feature Flags

Feature	Crate	Effect
fits-ingest	cal-io	Adds pub mod ingest wrapping zarr-fits-core
fits-ingest	cal-cli	Enables -f/--fits-folder and --zarr-cache flags
fits-ingest	cal-py	Exposes convert_fits_folder() to Python

Without fits-ingest, the workspace has zero C library dependencies — everything compiles with a standard Rust toolchain.