Overview#

System Context#

The calibration engine sits at the heart of the CCAT data reduction chain, between raw data ingestion and science-ready reduction:

        graph LR
  FITS[FITS files<br/>telescope] -->|zarr-fits| L0[L0 Zarr<br/>raw counts]
  L0 -->|calibrate| L1[L1 Zarr<br/>T_A* spectra]
  L1 -->|reduction_pipeline| L2[L2 Zarr<br/>gridded maps]

  style L0 fill:#e3f2fd
  style L1 fill:#e8f5e9
  style L2 fill:#fff3e0

The engine accepts L0 Zarr stores produced by the FITS-to-Zarr converter and writes calibrated L1 Zarr stores consumed by the downstream reduction pipeline (baseline subtraction, flagging, gridding, mapping).

What It Does#

  1. Ingest — reads raw 5D count arrays and metadata from L0 Zarr stores (optionally converting FITS folders on the fly)

  2. Calibrate — applies the heterodyne calibration equation using HOT/COLD loads, atmospheric model, and receiver parameters to produce antenna temperature $T_A^*$

  3. Write — stores calibrated spectra, system temperature, flags, and quality metrics in L1 Zarr stores

Workspace#

Seven-crate Cargo workspace:

Crate

Purpose

C deps

Feature gate

cal-schema

Shared L0/L1 schema constants (zero dependencies)

none

cal-core

Pure calibration math — RJ correction, gamma, Tsys, PWV fitting, modes

none

cal-io

Zarr I/O, pipeline orchestration, optional FITS ingest

cfitsio (optional)

fits-ingest

cal-cli

calibrate binary (clap CLI)

cfitsio (optional)

fits-ingest

cal-py

PyO3 Python bindings

cfitsio (optional)

fits-ingest

zarr-fits-core

FITS → Zarr v3 conversion library

cfitsio

zarr-fits-cli

Standalone zarr-fits conversion binary

cfitsio

Build Profiles#

Profile

Binaries

C deps

Use case

Pure Zarr (default)

calibrate

none

Production calibration from pre-converted Zarr stores

Full (fits-ingest)

calibrate, zarr-fits

libcfitsio-dev

Development, ingestion, end-to-end processing

Supported Telescopes#

Telescope support is driven by YAML profiles (see Telescope Profiles):

  • CCAT/FYST — 200–1200 GHz, Atacama site (5612 m)

  • SOFIA/upGREAT — 1000–5000 GHz, airborne (13 km)

  • Generic — fallback for any FITS data with standard keywords

Parity Validation#

The engine is validated against the legacy kalibrate (C++/Fortran) pipeline across 21 datasets covering all production modes (TP, OTF, chopped, beam-switch, gain interpolation, multi-frontend). The external parity suite (calibrate_parity_tests/) reports 16 PASS, 0 FAIL with T_sys deviations between 0.4% and 2.7%.

All deviations are below the acceptance threshold and arise from controlled differences (f64 vs f32 truncation in the legacy code).

The workspace contains 168 unit/integration tests (cargo test --workspace).