============
Data Formats
============

The calibration engine reads L0 Zarr stores and writes L1 Zarr stores.
Both use Zarr v3 with zstd compression and vlen-utf8 string arrays.

Data Levels
-----------

.. list-table::
   :header-rows: 1
   :widths: 10 25 30 35

   * - Level
     - Content
     - Producer
     - Consumer
   * - **L0**
     - Raw counts + metadata
     - zarr-fits-core
     - calibrate (cal-io)
   * - **L1**
     - Calibrated :math:`T_A^*` spectra
     - calibrate (cal-io)
     - reduction_pipeline
   * - **L1b**
     - Baseline-subtracted, flagged
     - reduction_pipeline
     - reduction_pipeline
   * - **L2**
     - Gridded maps
     - reduction_pipeline
     - Science users

L0 Zarr Structure
-----------------

.. code-block:: text

   session.zarr/
   ├── zarr.json                          # Zarr v3 root metadata
   ├── scan_025650/
   │   ├── source/
   │   │   ├── data_5d     [C, D, R, A, S]   i32  raw counts
   │   │   ├── sobsmode    [S]               str  ON/OFF/OTF-ON/OTF-OFF
   │   │   ├── mjd         [S]               f64  Modified Julian Date
   │   │   ├── exptime     [S]               f32  integration time (s)
   │   │   ├── elevation   [S]               f32  elevation (rad)
   │   │   ├── azimuth     [S]               f32  azimuth (rad)
   │   │   ├── pamb        [S]               f32  ambient pressure (Torr)
   │   │   ├── tamb        [S]               f32  ambient temperature (K)
   │   │   ├── signal_freq [S]               f64  signal sideband freq (Hz)
   │   │   ├── image_freq  [S]               f64  image sideband freq (Hz)
   │   │   ├── freq_res    [S]               f64  channel width (Hz)
   │   │   ├── freq_off    [S]               f64  frequency offset (Hz)
   │   │   ├── ref_channel [S]               f32  reference channel index
   │   │   ├── lloadsn     [S]               i32  last-load scan number
   │   │   ├── otf_lon     [S]               f64  OTF longitude offset (deg)
   │   │   ├── otf_lat     [S]               f64  OTF latitude offset (deg)
   │   │   ├── frontend_backends [S]         str  pixel IDs (e.g. HFAV_PX00)
   │   │   └── raw_fits_headers  [S]         str  original FITS headers (JSON)
   │   └── calibration/
   │       ├── data_5d     [C, D, R, A, S]   i32  HOT/COLD counts
   │       ├── sobsmode    [S]               str  HOT/COL/COLD/SKY
   │       ├── thot        [S]               f32  hot load temperature (K)
   │       ├── tcold       [S]               f32  cold load temperature (K)
   │       └── ...                           (same coords as source)
   └── scan_025651/
       └── ...

5D Array Layout
~~~~~~~~~~~~~~~

The primary data array has shape ``[C, D, R, A, S]``:

.. list-table::
   :header-rows: 1
   :widths: 10 25 65

   * - Axis
     - Name
     - Description
   * - C
     - Channels
     - Spectral channels (typically 1024–16384)
   * - D
     - Dumps
     - Time samples within a subscan
   * - R
     - Receivers
     - Pixels within an array (e.g. 7 for HFA)
   * - A
     - Arrays
     - Frontend-backend combinations (e.g. HFA, LFA, 4G1–4G4)
   * - S
     - Subscans
     - Observing phases (ON, OFF, HOT, COLD, etc.)

Missing dumps are padded with ``i32::MIN`` (converted to ``NaN`` on read).

L1 Zarr Structure
-----------------

.. code-block:: text

   l1_output/
   ├── zarr.json                          # cal_schema_version, cal_engine_version
   └── scan_025650/
       ├── spectra      [C, D, R, A, S]   f64  calibrated T_A* (K)
       ├── t_sys        [C, R, A, S]      f64  system temperature (K)
       ├── t_int        [S]               f64  integration time (s)
       ├── t_rec_ssb    [C, R, A]         f64  receiver temperature SSB (K)
       ├── gamma        [C, R, A]         f64  gain calibration factor
       ├── tau_signal   [C]               f64  zenith opacity, signal (Np)
       ├── tau_image    [C]               f64  zenith opacity, image (Np)
       ├── t_sky        [C, R, A]         f64  sky temperature (K)
       ├── flags        [C, D, R, A, S]   u16  quality bitmask
       ├── signal_freqs [C]               f64  signal frequencies (Hz)
       ├── image_freqs  [C]               f64  image frequencies (Hz)
       ├── otf_lon      [D, S]            f64  OTF longitude (deg, if OTF)
       ├── otf_lat      [D, S]            f64  OTF latitude (deg, if OTF)
       └── otf_airmass  [D, S]            f64  per-dump airmass (if OTF)

Scan-level attributes include ``scan_number``, ``source``, ``instmode``,
``rest_freq_hz``, ``pwv_mm``, ``cal_strategy``, ``ref_strategy``, and QA
metrics.

Schema Constants
----------------

All array and group names are defined in the ``cal-schema`` crate
(``crates/cal-schema/src/lib.rs``), the single source of truth for both
the FITS→Zarr writer and the calibration reader.