Deployment#

✓
Documentation Verified Last checked: 2026-03-07 Reviewer: Christof Buchbender

The Workflow Manager is deployed as four Docker containers managed via Docker Compose in the system-integration repository.

Container Architecture#

┌─────────────────────────┐  ┌─────────────────────────┐
│  trigger-manager        │  │  workflow-manager        │
│  Poll pipelines,        │  │  Build commands,         │
│  evaluate gaps           │  │  submit HPC jobs          │
└─────────────────────────┘  └─────────────────────────┘

┌─────────────────────────┐  ┌─────────────────────────┐
│  result-manager         │  │  celery-worker           │
│  Collect outputs,       │  │  Execute staging,        │
│  track lineage           │  │  HPC, result tasks        │
└─────────────────────────┘  └─────────────────────────┘

All containers use the same image: ghcr.io/ccatobs/workflow-manager:develop (staging) or :main (production).

Docker Compose#

The compose file lives in system-integration: docker-compose.staging.workflow-manager.yml

Starting the services:

cd /path/to/system-integration
docker compose -f docker-compose.staging.workflow-manager.yml up -d

Viewing logs:

docker compose -f docker-compose.staging.workflow-manager.yml logs -f workflow_trigger_manager

Required Environment Variables#

All containers need:

  • ENV_FOR_DYNACONF — environment selection

  • CCAT_WORKFLOW_MANAGER_REDIS_PASSWORD — Redis authentication

  • CCAT_WORKFLOW_MANAGER_REDIS_HOST — Redis host

  • CCAT_WORKFLOW_MANAGER_PROCESSING_LOCATION_ID — which location to manage

  • CCAT_OPS_DB_DATABASE_POSTGRESQL_* — database connection

Backend-specific containers additionally need:

  • CCAT_WORKFLOW_MANAGER_HPC_BACKEND — backend type

  • CCAT_WORKFLOW_MANAGER_K8S_NAMESPACE — for Kubernetes backend

  • CCAT_WORKFLOW_MANAGER_PIPELINE_BASE_DIR — pipeline workspace root

Volume Mounts#

Containers that need filesystem access require volume mounts:

  • /srv/pipelines:/data/pipelines — pipeline workspace, outputs, logs

  • /opt/redis-certs/develop:/etc/redis/certs:ro — Redis TLS certificates

CLI Commands#

Each container runs a single CLI command:

ccat_workflow_manager trigger-manager       # Start trigger evaluation loop
ccat_workflow_manager workflow-manager       # Start workflow orchestration loop
ccat_workflow_manager result-manager         # Start result collection loop
ccat_workflow_manager celery-worker <name> [-c <concurrency>]

CI/CD#

GitHub Actions builds and pushes the Docker image on push to develop or main:

  1. CI: ruff check + pytest

  2. Build: Docker image → GHCR (ghcr.io/ccatobs/workflow-manager:<branch>)

Staging deploys automatically from the develop branch.