Configuration#

✓
Documentation Verified Last checked: 2026-03-07 Reviewer: Christof Buchbender

The Workflow Manager uses Dynaconf for configuration management, with the prefix CCAT_WORKFLOW_MANAGER_. This follows the same pattern as data-transfer and ops-db.

Configuration Layers#

Settings are resolved in priority order (highest first):

  1. Environment variables — CCAT_WORKFLOW_MANAGER_*

  2. Environment-specific sections in settings.toml (selected by ENV_FOR_DYNACONF)

  3. Default section in settings.toml

Environment Selection#

Set ENV_FOR_DYNACONF to select the active environment:

  • localdev — local development

  • staging — staging deployment

  • production — production deployment

Core Settings#

Setting

Default

Description

HPC_BACKEND

"kubernetes"

HPC backend: kubernetes, slurm, or local

PIPELINE_BASE_DIR

"/data/pipelines"

Root directory for pipeline workspace, output, and logs

PROCESSING_LOCATION_ID

1

DataLocation ID this instance manages. Only pipelines assigned to this location are evaluated.

K8S_NAMESPACE

"ccat-workflows"

Kubernetes namespace for job submission

SLURM_PARTITION

""

SLURM partition name

SIF_CACHE_DIR

"/data/sif-cache"

Directory for cached Apptainer SIF images

Redis & Database#

The Workflow Manager shares Redis with data-transfer. Database connection is provided by the ccat_ops_db package.

Setting

Default

Description

REDIS_HOST

"localhost"

Redis host

REDIS_PORT

6379

Redis port

REDIS_PASSWORD

""

Redis password

Example Configurations#

Local development:

export ENV_FOR_DYNACONF=localdev
export CCAT_WORKFLOW_MANAGER_HPC_BACKEND=local
export CCAT_WORKFLOW_MANAGER_PIPELINE_BASE_DIR=/tmp/pipelines
export CCAT_WORKFLOW_MANAGER_PROCESSING_LOCATION_ID=1

Staging (Docker Compose):

environment:
  - ENV_FOR_DYNACONF=staging
  - CCAT_WORKFLOW_MANAGER_HPC_BACKEND=kubernetes
  - CCAT_WORKFLOW_MANAGER_K8S_NAMESPACE=ccat-workflows-staging
  - CCAT_WORKFLOW_MANAGER_PIPELINE_BASE_DIR=/data/pipelines
  - CCAT_WORKFLOW_MANAGER_PROCESSING_LOCATION_ID=1

Production (SLURM):

export ENV_FOR_DYNACONF=production
export CCAT_WORKFLOW_MANAGER_HPC_BACKEND=slurm
export CCAT_WORKFLOW_MANAGER_SLURM_PARTITION=science
export CCAT_WORKFLOW_MANAGER_PIPELINE_BASE_DIR=/data/pipelines
export CCAT_WORKFLOW_MANAGER_PROCESSING_LOCATION_ID=2

Settings File#

The settings file lives at ccat_workflow_manager/config/settings.toml:

[default]
HPC_BACKEND = "kubernetes"
K8S_NAMESPACE = "ccat-workflows"
SIF_CACHE_DIR = "/data/sif-cache"
PIPELINE_BASE_DIR = "/data/pipelines"
PROCESSING_LOCATION_ID = 1

[staging]
K8S_NAMESPACE = "ccat-workflows-staging"

[production]
HPC_BACKEND = "slurm"

[localdev]
HPC_BACKEND = "local"
PIPELINE_BASE_DIR = "/tmp/pipelines"