Sky Diagnostics (SKYDIFF / SKYCHOPDIFF)#

Two optional diagnostics, ported from kalibrate, that probe sky and instrument stability independently of the source signal. Both are written into the L1 Zarr store alongside the calibrated spectra and are off by default.

Diagnostic

Flag

Question it answers

SKYDIFF

--skydiff <N>

How does the observed sky+receiver spectrum drift between calibration cycles (standing waves, gain drift)?

SKYCHOPDIFF

--skychopdiff

Are there baseline differences within one OFF between chop phases that the cross-scan SKYDIFF averages away?

SKYDIFF (--skydiff <N>)#

Differences the calibrated observed sky-hot spectrum

S_H_OBS = (OFF - HOT) / gamma        [Kelvin]

(kalibrate’s deltat_obs) against the N most recent previous OFFs. Because both entries are calibrated to Kelvin before differencing, the Rayleigh-Jeans hot-load correction (a few K) survives; differencing raw counts would bury it under the ~1e5-scale count level.

Cadence and history rules (matching kalibrate’s per-cal-cycle AOBS):

  • one history entry per OFF subscan; on chopped / beam-switch scans one entry per reduction group (the group’s own-beam OFF, or the 50/50 cross-beam blend when --dbs-coupling is active),

  • FIFOs are keyed per band and per backend/pixel layout — scans observing different pixels never mix (kalibrate keys its history per spectrometer),

  • the history needs at least two OFFs, so diffs appear from the second qualifying scan onward,

  • the cross-scan history requires in-order processing: when --skydiff is active the pipeline automatically runs scans serially instead of in parallel.

Output layout (per scan that produced diffs):

/scan_NNNNNN/skydiff/spectrum_K          [C, R, A] float64, K = 0..n_diffs-1
/scan_NNNNNN/band_B/skydiff/spectrum_K   (multi-frontend scans)

Group attributes are parallel arrays indexed like the spectra: n_diffs, current_scan, current_mjds, current_offs, ref_scans, ref_mjds, ref_offs — i.e. spectrum_K is current OFF (current_offs[K]) minus reference OFF (ref_offs[K] of ref_scans[K]).

SKYCHOPDIFF (--skychopdiff)#

For chopped / beam-switch scans only: the calibrated difference of the sky’s chop/dump phase 0 against every later phase j within each reduction group,

spectrum_j = t_cal * (sky_0 - sky_j) / (hot - cold) / tr_s

(kalibrate -m skychopdiff, gamma-factor branch). tr_s is the per-channel signal-band atmospheric transmission of the group’s tagged ON — the same factor the main calibration applies. kalibrate’s cal->data calibration factor bakes this atmospheric correction in; omitting it scales every channel by a flat t_cal where kalibrate applies the curved atmospheric line shape, which capped per-dump parity at corr ~0.84 before the factor was included. Because the transmission is only known after the atmosphere stage, SKYCHOPDIFF is computed after it (see Pipeline).

By design calibrate does not build the intrinsic chopped A+B sky combination by default — each beam’s sky is used individually so beam-specific issues stay fixable downstream. --dbs-coupling blends the opposite-beam partner OFF 50/50, reproducing legacy kalibrate’s default dbs_adhesive association for parity runs (per-dump corr

0.9998 against kalibrate).

Output layout:

/scan_NNNNNN/skychopdiff/spectrum_<on>_<j>   [C, R, A] float64

<on> is the positional subscan index of the group’s tagged ON, <j> the chop position (1..D-1). Group attributes: on_indices, on_mjds, n_spectra_per_on. Bad channels and opaque channels (tr_s <= 0) are NaN.

Running#

Basic SKYDIFF over a session (diff each OFF against the 3 most recent):

calibrate -i session.zarr --output l1/ --atm-table atm.catm \
  --skydiff 3

Chopped data with both diagnostics, in kalibrate-comparison trim:

calibrate -i session.zarr --output l1/ --atm-table atm.catm \
  --physics kalibrate-compat --foeff 0.97 \
  --skydiff 3 --skychopdiff --dbs-coupling

Drop --dbs-coupling (and usually --physics kalibrate-compat) for production runs: each beam’s sky then stays separate by design.

The kalibrate equivalents are -m skydiff=N -m skychopdiff=1 (SDFITS rows SKY-DIFF / SKYCHOPDIFF, requires -m disable_class_output=1).

Reading the output:

import zarr

scan = zarr.open("l1/", mode="r")["scan_030604"]

# SKYDIFF: pairing metadata lives in the group attrs
sd = scan["skydiff"]
for k in range(sd.attrs["n_diffs"]):
    spec = sd[f"spectrum_{k}"][:, 0, 0]           # [C] for pixel (0, 0)
    print(k, "OFF", sd.attrs["current_offs"][k],
          "- scan", sd.attrs["ref_scans"][k], "OFF", sd.attrs["ref_offs"][k])

# SKYCHOPDIFF: one spectrum per (tagged ON, chop position j)
scd = scan["skychopdiff"]
for on, n in zip(scd.attrs["on_indices"], scd.attrs["n_spectra_per_on"]):
    for j in range(1, n + 1):
        spec = scd[f"spectrum_{on}_{j}"][:, 0, 0]

Parity against kalibrate is regression-tested in calibrate_parity_tests (SKYDIFF_TP / SKYDIFF_BS / SKYDIFF_CHOPPED, compare: skydiff).