CCAT System Integration Documentation#

Overview#

This documentation covers the CCAT System Integration, including:

  • Docker Compose configuration and deployment

  • Secrets management via Ansible Vault and .env provisioning

  • Configuration management with Dynaconf

  • GitHub PAT (Personal Access Token) rotation procedures

  • Backup and restore procedures

  • Monitoring with Grafana, Prometheus, and Loki

  • Service orchestration across multiple nodes

Key Components#

Infrastructure Services:

  • PostgreSQL database with replication

  • Redis cache with TLS

  • InfluxDB for time-series data

  • MinIO object storage

  • Grafana dashboards

  • Prometheus monitoring

  • Loki log aggregation

Deployment Configurations:

  • Local development (docker-compose.local.yml)

  • Staging environment (docker-compose.staging.yml)

  • Production nodes:

    • Database node (docker-compose.production.db.yml)

    • Data processing node (docker-compose.production.data.yml)

    • Input node (docker-compose.production.input.yml)

    • Reuna VM (docker-compose.production.reuna-vm.yml)

Documentation Contents#

source/getting-started source/configuration source/deployment source/secrets-management source/monitoring source/backup-restore source/pat-rotation source/matrix source/proposal-update-workflow-full source/proposal-update-workflow-small-team source/background/index

Getting Started#

For setup instructions, see the project README and the Getting Started guide.

Indices and tables#

  • genindex

* search ======= System Integration & Deployment ================================

The system-integration repository contains Docker Compose configurations, Kubernetes manifests, and supporting infrastructure for deploying the complete CCAT data center stack across different environments.

Deployment Environments#

Local Development

  • docker-compose.localdev.yml - Infrastructure containers (postgres, redis, grafana, etc.)

  • process-compose.localdev.yml - Application processes with hot-reload via process-compose

  • docker-compose.dev.yml - Alternative: full stack in Docker with volume-mounted code

Staging

  • docker-compose.staging.yml - Staging environment

  • docker-compose.staging.batleth.yml - Batleth staging server

  • cologne-staging-k8s/ - Kubernetes configuration for Cologne staging

Production

  • docker-compose.production.*.yml - Production deployment configurations

  • Separated into database, data transfer, and input components

Infrastructure Components#

Core Services

  • PostgreSQL: Operations database with replication support

  • Redis: Caching and session management

  • MinIO: S3-compatible object storage

  • InfluxDB: Time-series metrics storage

Monitoring & Observability

  • Prometheus: Metrics collection and alerting

  • Grafana: Dashboards and visualization

  • Loki: Log aggregation

  • Promtail: Log collection agent

Supporting Services

  • Nginx: Reverse proxy and load balancing

  • Celery: Distributed task queue for background jobs

Directory Structure#

postgres/

PostgreSQL initialization scripts and configuration

grafana/

Grafana dashboards and provisioning configuration

prometheus/

Prometheus monitoring configuration

loki-config/ & promtail-config/

Log aggregation configuration

jenkins_pipelines/

CI/CD pipeline definitions

tools/

Deployment and management scripts

Deployment Guides#

Kubernetes Deployments#

The cologne-staging-k8s/ directory contains Kubernetes manifests for deploying on the Cologne cluster:

  • Deployment configurations

  • Service definitions

  • Ingress routing

  • Persistent volume claims

See cologne-staging-k8s/README.md for detailed deployment instructions.

Docker Compose Deployments#

Docker Compose files are organized by environment and component:

  1. Choose your environment: local, staging, or production

  2. Select components: database, data transfer, input processing

  3. Run: docker-compose -f <config-file> up -d

Initialization Scripts#

init_ops_db_entrypoint.sh

Initializes the operations database schema and loads seed data

init-influxdb.sh

Sets up InfluxDB databases and retention policies

init-minio.sh

Configures MinIO buckets and access policies

Monitoring#

The monitoring stack provides comprehensive observability:

  • System Metrics: CPU, memory, disk, network via Prometheus

  • Application Metrics: Custom metrics from Python services

  • Logs: Centralized logging with Loki

  • Dashboards: Pre-configured Grafana dashboards for all services

Access Grafana at the configured endpoint to view dashboards and alerts.