Overview#

System Context#

The calibration engine sits at the heart of the CCAT data reduction chain, between raw data ingestion and science-ready reduction:

        graph LR
  FITS[FITS files<br/>telescope] -->|zarr-fits| L0[L0 Zarr<br/>raw counts]
  L0 -->|calibrate| L1[L1 Zarr<br/>T_A* spectra]
  L1 -->|reduction_pipeline| L2[L2 Zarr<br/>gridded maps]

  style L0 fill:#e3f2fd
  style L1 fill:#e8f5e9
  style L2 fill:#fff3e0

The engine accepts L0 Zarr stores produced by the FITS-to-Zarr converter and writes calibrated L1 Zarr stores consumed by the downstream reduction pipeline (baseline subtraction, flagging, gridding, mapping).

What It Does#

  1. Ingest — reads raw 5D count arrays and metadata from L0 Zarr stores (optionally converting FITS folders on the fly)

  2. Calibrate — applies the heterodyne calibration equation using HOT/COLD loads, atmospheric model, and receiver parameters to produce antenna temperature \(T_A^*\)

  3. Write — stores calibrated spectra, system temperature, flags, and quality metrics in L1 Zarr stores

Workspace#

Seven-crate Cargo workspace:

Crate

Purpose

C deps

Feature gate

cal-schema

Shared L0/L1 schema constants (zero dependencies)

none

cal-core

Pure calibration math — RJ correction, gamma, Tsys, PWV fitting, modes

none

cal-io

Zarr I/O, pipeline orchestration, optional FITS ingest

cfitsio (optional)

fits-ingest

cal-cli

calibrate binary (clap CLI)

cfitsio (optional)

fits-ingest

cal-py

PyO3 Python bindings

cfitsio (optional)

fits-ingest

zarr-fits-core

FITS → Zarr v3 conversion library

cfitsio

zarr-fits-cli

Standalone zarr-fits conversion binary

cfitsio

Build Profiles#

Profile

Binaries

C deps

Use case

Pure Zarr (default)

calibrate

none

Production calibration from pre-converted Zarr stores

Full (fits-ingest)

calibrate, zarr-fits

libcfitsio-dev

Development, ingestion, end-to-end processing

Supported Telescopes#

Telescope support is driven by YAML profiles (see Telescope Profiles):

  • CCAT/FYST — 200–1200 GHz, Atacama site (5612 m)

  • SOFIA/upGREAT — 1000–5000 GHz, airborne (13 km)

  • Generic — fallback for any FITS data with standard keywords

Parity Validation#

The engine is validated against the legacy kalibrate (C++/Fortran) pipeline across 7 datasets covering all production modes:

Dataset

Mode

Pixels

Max Dev

Status

HFAV_OTF

OTF TP

7 HFA

0.004%

PASS

HFAV_OI_BLANK_TEST

OTF TP

7 HFA

0.034%

PASS

OFF_EXCLUDE_TEST

OTF TP

1 LFA

0.001%

PASS

LFAV_HONEYCOMB

OTF-OFF

1 LFA

0.0004%

PASS

HFAV_4G_MISSING_DATA

multi-band

11

0.635%

PASS

HFA_4G_LOAD_TABLE

multi-band + load_temp

11

0.642%

PASS

SIMULATED_DATA_HFA_4G

multi-band (Tier 3)

5

0.034%

PASS

All deviations are below the 1% acceptance threshold and arise from controlled differences (f64 vs f32 truncation in the legacy code).