## Definition and guidance for the interface between instrument control software, operations database (ops-db) and scheduler The purpose of this document is to have a common understanding of the interface and procedure in observing and data delivery on the mountain, and to catch a potential oversight in the implementation early enough. We are still in the development phase, so things may change in the coming months but we should communicate. ### Relevant repositories (accessible by CCAT consortium members) + operations database (ops-db) https://github.com/ccatobs/ops-db Follow the "First time setup on a local computer" section of the README of the repository. The default branch is "develop". + data-center https://github.com/ccatobs/data-center This is only needed locally if you are setting-up the ops-db using a different branch or locally modified content (this is because the ops-db is built using the content stored in the data-center repository). The default branch is "develop". + obs_implementation https://github.com/ccatobs/obs_implementation This is the repository for the scheduler, as well as for creating ObsUnit content (see the introduction in the linked documentation from its README). Follow its README for installation (and the additional step for running the scheduler under "Use cases"). The default branch is "main". ### Brief explanation of the scripts that are relevant to the interface and mentioned below + Under obs_implementation repository + *fystplan/lib/setup_available_opsdb.py* Modify the availability of ObsUnits, instruments and modules in obs-db. It can also register the scheduler result file to create corresponding ExecutedObsUnits for testing. + *fystplan/lib/interact_opsdb.py* Collections of functions that query the ops-db. They may be converted to API later. Therefore, we try to keep all direct ops-db communication used under the fystplan package in this script to make the conversion to API easier. + *fystplan/scheduling/observer_interface.py* This is not a real observer interface yet, but a collection of elements that should run behind the observer interface, such as reading the scheduler results and printing the ObsUnit information, plotting the elevations and sun avoidance of the alternative ObsUnits, etc. + *fystplan/PrimeCam_planning/create_PrimeCam_sourcecatalog.py Tools to create initial input files for the ops-db based on input from the science team for the ModCam and PrimeCam observing programs. + Under py_kosma_scripts py_kosma_scripts is the part of the kosma_software package (CHAI instrument control and observation software) on a Cologne git repository. This particular part of py_kosma_scripts is specialized for ops-db communication and worth sharing across the instrument, so we should establish an access procedure from the PrimeCam team. + py_kosma_scripts/interact_opsdb.py Collections of functions that query the ops-db from the instrument software. Same philosophy as in fystplan. + Other scripts in py_kosma_scripts/ Python support scripts that can be called from kosma_software/kosma_scripts (bash). ### Definition of ObsUnits to be considered by the scheduler The following classes in the ops-db have “available” flags. 1. ObsUnit 2. InstrumentModule (e.g. f280 for modcam, L and H for miniCHAI etc.) 3. Instrument (modcam, primecam, miniCHAI, CHAI) Each ObsUnit has a primary_instrument_module_configuration (science driver), which consists of an instrument module, which is then associated with an instrument. The scheduler will consider all ObsUnits with all three available flags set to 1, namely (1) the ObsUnit itself, (2) the instrument module set as primary_instrument_module_configuration, and (3) the instrument associated with the primary_instrument_module_configuration. ### At the start of the campaign As part of starting an instrument campaign, the available flags of the instrument and modules must to be set properly. The script is to do this is *setup_available_opsdb.py*. For example, setup_available_opsdb --modcam_es Available flags of individual modules can also be set separately (see -h for mode explanation). ### Any time the available module changes Use the same script with -m to change the availability. ### Scheduler output At each run of the scheduler (every X-min, TBD), it outputs the best suggested ObsUnit and several (default 5, can be changed as argument when running the scheduler) alternatives to the Redis as hashes. + **scheduler:next_best_obs_unit** has + timestamp + start_time + value + **scheduler:next_obs_unit_alternatives** has + timestamp + start_time + alt_1 + alt_2 + alt_3 + alt_4 + alt_5 The content of value and alt_X is the name of the ObsUnit. The timestamp is the time when the information was written, start_time is the planned start time of the next schedule (with the rough knowledge of when the currently running ObsUnit ends, see below). The reason for using the ObsUnit name instead of the ID is the scheduler internal reason and human readability, but if there is a strong reason to use the ID instead, we can change it. Note that the ObsUnit name must to be unique anyway, since the scheduler uses the names internally. This uniqueness must be guaranteed by any script that adds or modifies the ObsUnit in the ops-db. ### Instrument software reads the scheduler output The above contents of redis can be read in python, e.g. key = "scheduler:next_best_obs_unit" redis.hgetall(key) see *observer_interface.py* for reading examples. It is recommended to check that the timestamp is not too old when reading. ### Instrument software reads the ObsUnit information Once the instrument software knows the ObsUnit name (default could be the one read above, but whatever that the observer decides), it can get the full information using the *get_obs_unit_from_name* function in *interact_opsdb.py*. There are a number of Mod-Cam / Prime-Cam observation parameters that have already been introduced in the ops-db as they were needed to develop the scheduler, but they need to be iterated with the actual instrument software. Each ObsUnit has the following attributes (not a complete list, only those with additional notes are listed). + source This gives the Source class. Sources have three types, fixed_source, constant_elevation_source, and solar_system_object. Fixed_source has a standard coordinates, solar_system_object has an eph_name that can be compatible with get_body in astropy, and a naif_id (can be used in case of an object that is not included in get_body). constant_elevation_source has a range of coordinates that should be covered. + obs_mode This gives the obs_mode class, which basically specifies the type of the scan pattern. + nominal_alt This attribute is only valid if the associated source is constant_elevation_source and is either a float for a fixed elevation (example is Deep56, 50 deg or 60 deg) or "var" to allow a flexible choice of elevation between min_alt and max_alt (example is GAMA-09). In case of "var", the scheduler calculates the optimal elevation using the RAmin of the source and the expected start time to judge if it is observable. This calculation is done by the _get_optimal_elevation function in obs_implementation/fystplan/scheduling/FysctScheduling.py. We should coordinate between the scheduler and the instrument software to use the same logic to calculate this optimal elevation at a given time. + observation_configuration This gives the observation_configuration class. The available attributes of observation_configuration are + azimuth_range This is only valid for constant_elevation_source and is pre-calculated based on the given range of coordinates and elevation. This is a JSON field with rising or setting specified, and the list of elevations and az_min and az_max given. The list contains only one elevation if nominal_alt above is one elevation, and a 1-degree increment list if nominal_alt is "var". This way the instrument software can select the azimuth range from these attributes once the elevation is set, instead of calculating it each time from the requested field coverage. + mapping_parameters (if observation_configuration is prime_cam_observation_configuration) This is a JSON field for entering any mapping/scan pattern parameters. For ease of scan-pattern simulation, the parameters of Pong patterns are currently registered. The scheduler uses this field (if the ObsUnit is not constant elevation scans) to infer the map size for the purpose of judging sun avoidance constraints. + version, history (if observation_configuration is prime_cam_observation_configuration) This is a field for book-keeping on the Prime-Cam side if needed. + CHAI has additional association to chai_tiling and chai_inpar_parameters for book-keeping all mapping parameters. ### When the instrument software starts the observations It is important to coordinate the way the instrument software reports the start and end of observations with the specified ObsUnit. The current scheme is to do this through the ExecutedObsUnit class. This information is used in the scheduler run in-between, to judge when the currently running observation ends and to propose the next schedule assuming the currently running observation will end successfully. Here are the details using CHAI as an example. 1. When we start a new observation, we create a new ExecutedObsUnit by specifying + The associated ObsUnit + start_time + status = 'running' These three attributes in ExecutedObsUnit are essential for the scheduler to recognize what is running now. The script used for CHAI is py_kosma_scripts/add_or_update_executed_obs_unit_at_start.py, the actual ops-db communication is done by the function add_executed_obs_unit or update_executed_obs_unit in py_kosma_scripts/interact_opsdb.py. An ExecutedObsUnit class also has optional JSON fields obs-info and obs-progress. These are for internal use by each instrument to record any useful information. For CHAI, we record a version number of "chai_inpar" (detailed observation parameters such as mapping size, scan speed, etc.), scan numbers in obs-info, and progress of OTF lines in obs-progress. 2. When the observation ends successfully, the status of the corresponding ExecutedObsUnit must be changed to any string containing "success", and the end_time should also be added. The scheduler counts the achieved hours of each ObsUnit by counting the number of ExecutedObsUnits with "success" in the status string. For CHAI we use py_kosma_scripts/update_progress_executed_obs_unit.py to update this information. ExecutedObsUnit also has fields of mean_pwv and mean_elevation which should provide useful information and they may be used in the future scheduler to calculate more sophisticated achievement (effective hours taking into account atmospheric transmission). But it is still TBD how to implement this (could be postponed to year 2). 3. If the observation is interrupted (e.g. software crash, observation stopped due to strong wind, etc.), the status of the corresponding ExecutedObsUnit needs to be changed. The exact choice of wording in the status field is up to the instrument, but it must be + not "running" + does not contain "success" Please also add end_time to ExecutedObsUnit. For CHAI we use "interrupted" (or "resumed-interrupted"). The script used is the same as the successful end above, and we implemented it to run before exiting the script with an error or Ctr-C. ### Instrument software registers the raw data files in ops-db All raw data files must be registered as RawDataFile in ops-db. The required information is + File name, path, and file type (e.g. fits, g3) + Size (in bytes, e.g. os.path.getsize) + checksum (can be generated by, e.g. ~~~ with open(full_path, "rb") as f: checksum = hashlib.sha256(f.read()).hexdigest() ~~~ + associated ObsUnit ID + associated InstrumentModuleConfiguration ID (because an ObsUnit has multiple instrument module configurations) ### Instrument software zips a group of raw data files and registers them as raw data packages in ops-db A raw data package is the unit of data stored in the final long term archive. It consists of a meaningful group of raw data files. For CHAI, we plan to group raw data files by ExecutedObsUnit. Each package should not exceed 50GB. There is no lower limit, because if it is too small for efficient data transfer, it is the responsibility of the data transfer task to further combine raw data packages into data transfer packages. The important thing is that this raw data package is the unit of data delivery with any data archive query later, and the creation of the raw data packages is still the responsibility of the instrument software. The required information is + File name, relative path + Size, checksum, associated ObsUnit ID + Creation time + RawDataFile list (pointing to RawDataFile entries) ### Science / instrument team updates the content of ops-db We still need to develop a set of functions (then APIs) to safely and efficiently add and update the contents of the ops-db once it is in operation. The following is the consideration we need to keep in mind. We should also keep the full log of all changing commands so that we can trace back if needed. + version/history These fields are introduced in ObsUnit, ChaiTiling, ChaiInparParameter, and PrimeCamObservationConfiguration. The idea is to record all changes in the history field as JSON (timestamp, variable changed, old value) and increment the version whenever it changes. This should make it easier to track a case like "what obs-parameters were used to perform this observation?". + PreScheduledSlot The start and end time of a specific ObsUnit can be registered as PreScheduledSlot to ensure that this ObsUnit has the highest priority during this time range. This is basically meant for WFS (but can be applied to any other ObsUnit if needed). Currently it reserves 24 hours in every 3 days from 2025 to 2028 (12 hours before midnight (lowest sun elevation) is on the rising side, 12 hours after midnight is on the setting side, with 15 min margin as overlap due to edge treatment), and simply rotates it among 6 ObsUnits (40 and 50 deg elevations times three different scan patterns). See create_PrimeCam_sourcecatalog.py for the exact calculations. Before the real observations, the actual schedule has to be provided by the Prime-Cam team. + phase This is one of the ObsUnit attributes, but it is included in the ObsUnit name to keep it unique. Currently, "CM" is commissioning, "ES_1MDL" is Early Science with Mod-Cam (1 module), "ES_4MDL" is Early Science with Prime-Cam (4 modules), "ES" is miniCHAI, "BS" is Baseline science for Prime-Cam and CHAI. Because the name is naturally split between CHAI and Prime-Cam campaigns, either side is free to change or add the name of the phase, but please do not forget to change the name of the ObsUnit (API could introduce a warning). + ObsUnit name As mentioned above, the ObsUnit name must be unique. When developing the script/API to update the contents of the ObsUnit, we should also remember that ObsUnit names are also used in the grouping field, so if the name changes, it needs to go through the whole database to make sure that the name in the grouping field is also updated accordingly. + Backup ObsUnits In some cases, e.g. Pong vs. constant elevation scans for ModCam observations, we may want a backup ObsUnit so that we can quickly switch between two possibilities after commissioning. We should prepare both and set the available flag to 0 for the backup ObsUnits, and prepare the script to switch them. This way we can also fully prepare the observation parameters for backup scan patterns.