# Sky Diagnostics (SKYDIFF / SKYCHOPDIFF)

Two optional diagnostics, ported from kalibrate, that probe sky and
instrument stability independently of the source signal. Both are
written into the L1 Zarr store alongside the calibrated spectra and
are off by default.

| Diagnostic | Flag | Question it answers |
|---|---|---|
| SKYDIFF | `--skydiff <N>` | How does the observed sky+receiver spectrum drift *between* calibration cycles (standing waves, gain drift)? |
| SKYCHOPDIFF | `--skychopdiff` | Are there baseline differences *within* one OFF between chop phases that the cross-scan SKYDIFF averages away? |

## SKYDIFF (`--skydiff <N>`)

Differences the calibrated **observed sky-hot spectrum**

```text
S_H_OBS = (OFF - HOT) / gamma        [Kelvin]
```

(kalibrate's `deltat_obs`) against the `N` most recent previous OFFs.
Because both entries are calibrated to Kelvin before differencing, the
Rayleigh-Jeans hot-load correction (a few K) survives; differencing raw
counts would bury it under the ~1e5-scale count level.

Cadence and history rules (matching kalibrate's per-cal-cycle AOBS):

- one history entry per OFF subscan; on chopped / beam-switch scans one
  entry per reduction group (the group's own-beam OFF, or the 50/50
  cross-beam blend when `--dbs-coupling` is active),
- FIFOs are keyed per **band and per backend/pixel layout** — scans
  observing different pixels never mix (kalibrate keys its history per
  spectrometer),
- the history needs at least two OFFs, so diffs appear from the second
  qualifying scan onward,
- the cross-scan history requires in-order processing: when
  `--skydiff` is active the pipeline automatically runs scans serially
  instead of in parallel.

Output layout (per scan that produced diffs):

```text
/scan_NNNNNN/skydiff/spectrum_K          [C, R, A] float64, K = 0..n_diffs-1
/scan_NNNNNN/band_B/skydiff/spectrum_K   (multi-frontend scans)
```

Group attributes are parallel arrays indexed like the spectra:
`n_diffs`, `current_scan`, `current_mjds`, `current_offs`,
`ref_scans`, `ref_mjds`, `ref_offs` — i.e. `spectrum_K` is
*current OFF (current_offs[K]) minus reference OFF (ref_offs[K] of
ref_scans[K])*.

## SKYCHOPDIFF (`--skychopdiff`)

For chopped / beam-switch scans only: the calibrated difference of the
sky's chop/dump phase 0 against every later phase `j` within each
reduction group,

```text
spectrum_j = t_cal * (sky_0 - sky_j) / (hot - cold) / tr_s
```

(kalibrate `-m skychopdiff`, gamma-factor branch). `tr_s` is the
per-channel signal-band atmospheric transmission of the group's tagged
ON — the same factor the main calibration applies. kalibrate's
`cal->data` calibration factor bakes this atmospheric correction in;
omitting it scales every channel by a flat t_cal where kalibrate
applies the curved atmospheric line shape, which capped per-dump parity
at corr ~0.84 before the factor was included. Because the transmission
is only known after the atmosphere stage, SKYCHOPDIFF is computed after
it (see {doc}`pipeline`).

By design calibrate does **not** build the intrinsic chopped A+B sky
combination by default — each beam's sky is used individually so
beam-specific issues stay fixable downstream. `--dbs-coupling` blends
the opposite-beam partner OFF 50/50, reproducing legacy kalibrate's
default `dbs_adhesive` association for parity runs (per-dump corr
> 0.9998 against kalibrate).

Output layout:

```text
/scan_NNNNNN/skychopdiff/spectrum_<on>_<j>   [C, R, A] float64
```

`<on>` is the positional subscan index of the group's tagged ON,
`<j>` the chop position (1..D-1). Group attributes: `on_indices`,
`on_mjds`, `n_spectra_per_on`. Bad channels and opaque channels
(`tr_s <= 0`) are NaN.

## Running

Basic SKYDIFF over a session (diff each OFF against the 3 most recent):

```bash
calibrate -i session.zarr --output l1/ --atm-table atm.catm \
  --skydiff 3
```

Chopped data with both diagnostics, in kalibrate-comparison trim:

```bash
calibrate -i session.zarr --output l1/ --atm-table atm.catm \
  --physics kalibrate-compat --foeff 0.97 \
  --skydiff 3 --skychopdiff --dbs-coupling
```

Drop `--dbs-coupling` (and usually `--physics kalibrate-compat`) for
production runs: each beam's sky then stays separate by design.

The kalibrate equivalents are `-m skydiff=N -m skychopdiff=1`
(SDFITS rows `SKY-DIFF` / `SKYCHOPDIFF`, requires
`-m disable_class_output=1`).

Reading the output:

```python
import zarr

scan = zarr.open("l1/", mode="r")["scan_030604"]

# SKYDIFF: pairing metadata lives in the group attrs
sd = scan["skydiff"]
for k in range(sd.attrs["n_diffs"]):
    spec = sd[f"spectrum_{k}"][:, 0, 0]           # [C] for pixel (0, 0)
    print(k, "OFF", sd.attrs["current_offs"][k],
          "- scan", sd.attrs["ref_scans"][k], "OFF", sd.attrs["ref_offs"][k])

# SKYCHOPDIFF: one spectrum per (tagged ON, chop position j)
scd = scan["skychopdiff"]
for on, n in zip(scd.attrs["on_indices"], scd.attrs["n_spectra_per_on"]):
    for j in range(1, n + 1):
        spec = scd[f"spectrum_{on}_{j}"][:, 0, 0]
```

Parity against kalibrate is regression-tested in
`calibrate_parity_tests` (SKYDIFF_TP / SKYDIFF_BS / SKYDIFF_CHOPPED,
`compare: skydiff`).