Site Configuration#

✓

Documentation Verified Last checked: 2025-11-12 Reviewer: Christof Buchbender

The ops-db-api uses site-aware configuration to automatically adapt its behavior based on deployment location (main site vs secondary site).

Overview #

Site Type determines how the API behaves:

MAIN site (Cologne): Direct database access, no buffering
SECONDARY site (Observatory): Buffered writes, replica reads

The site configuration is managed by the SiteConfig class:

class SiteType(str, Enum):
    """Site types for distributed database architecture"""

    MAIN = "main"
    SECONDARY = "secondary"


class SiteConfig:
    """Site configuration manager"""

    def __init__(self):
        self.site_name = ccat_ops_db_api_settings.get("site_name", "institute")
        self.site_type = SiteType(ccat_ops_db_api_settings.get("site_type", "main"))
        self.critical_operations_buffer = ccat_ops_db_api_settings.get(
            "critical_operations_buffer", True
        )

        # Main database configuration (for transaction buffering)
        self.main_db_type = ccat_ops_db_api_settings.get("main_db_type", "postgresql")
        self.main_db_host = ccat_ops_db_api_settings.get("main_db_host", "localhost")
        self.main_db_port = ccat_ops_db_api_settings.get("main_db_port", "5432")
        self.main_db_user = ccat_ops_db_api_settings.get(
            "main_db_user", "ccat_ops_user"
        )
        self.main_db_password = ccat_ops_db_api_settings.get(
            "main_db_password", "ccat_ops_password"
        )
        self.main_db_name = ccat_ops_db_api_settings.get("main_db_name", "ccat_ops_db")

        # Local database configuration (for direct access)
        self.local_db_type = ccat_ops_db_api_settings.get("local_db_type", "postgresql")
        self.local_db_host = ccat_ops_db_api_settings.get("local_db_host", "localhost")
        self.local_db_port = ccat_ops_db_api_settings.get("local_db_port", "5432")
        self.local_db_user = ccat_ops_db_api_settings.get(
            "local_db_user", "ccat_ops_user"
        )
        self.local_db_password = ccat_ops_db_api_settings.get(
            "local_db_password", "ccat_ops_password"
        )
        self.local_db_name = ccat_ops_db_api_settings.get(
            "local_db_name", "ccat_ops_db"
        )

        # Transaction buffering configuration
        self.transaction_buffer_size = ccat_ops_db_api_settings.get(
            "transaction_buffer_size", 1000
        )
        self.transaction_retry_attempts = ccat_ops_db_api_settings.get(
            "transaction_retry_attempts", 3
        )
        self.transaction_retry_delay = ccat_ops_db_api_settings.get(
            "transaction_retry_delay", 5
        )
        self.background_processing_interval = ccat_ops_db_api_settings.get(
            "background_processing_interval", 1.0
        )

        # LSN tracking configuration
        self.lsn_tracking_enabled = ccat_ops_db_api_settings.get(
            "lsn_tracking_enabled", True
        )
        self.lsn_check_interval = ccat_ops_db_api_settings.get(
            "lsn_check_interval", 0.1
        )
        self.lsn_timeout = ccat_ops_db_api_settings.get("lsn_timeout", 30)

    @property
    def is_main_site(self) -> bool:
        """Check if this is the main site"""
        return self.site_type == SiteType.MAIN

    @property
    def is_secondary_site(self) -> bool:
        """Check if this is a secondary site"""
        return self.site_type == SiteType.SECONDARY

Configuration Sources #

Settings are loaded from multiple sources (in order of precedence):

Environment variables (highest priority)
``.env`` file (development)
``config/settings.toml`` (application defaults)

Example .env File #

# Site Identity
SITE_NAME=observatory
SITE_TYPE=secondary  # "main" or "secondary"

# Main Database (for writes)
MAIN_DB_TYPE=postgresql
MAIN_DB_HOST=main-db.example.com
MAIN_DB_PORT=5432
MAIN_DB_USER=ccat_ops_user
MAIN_DB_PASSWORD=secure_password
MAIN_DB_NAME=ccat_ops_db

# Local Database (for reads)
LOCAL_DB_TYPE=postgresql
LOCAL_DB_HOST=localhost
LOCAL_DB_PORT=5432
LOCAL_DB_USER=ccat_ops_user
LOCAL_DB_PASSWORD=secure_password
LOCAL_DB_NAME=ccat_ops_db

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
REDIS_PASSWORD=

# Transaction Buffering
CRITICAL_OPERATIONS_BUFFER=true
TRANSACTION_BUFFER_SIZE=1000
TRANSACTION_RETRY_ATTEMPTS=3
TRANSACTION_RETRY_DELAY=5
BACKGROUND_PROCESSING_INTERVAL=1.0

# LSN Tracking
LSN_TRACKING_ENABLED=true
LSN_CHECK_INTERVAL=0.1
LSN_TIMEOUT=30

# Authentication
SECRET_KEY=your-secret-key-change-in-production
GITHUB_CLIENT_ID=your-github-oauth-client-id
GITHUB_CLIENT_SECRET=your-github-oauth-client-secret

Example settings.toml #

[default]
site_name = "institute"
site_type = "main"

[default.database]
main_db_host = "localhost"
main_db_port = 5432
local_db_host = "localhost"
local_db_port = 5432

[default.redis]
host = "localhost"
port = 6379

[default.transaction_buffering]
buffer_size = 1000
retry_attempts = 3
retry_delay = 5
background_processing_interval = 1.0

Site Types #

MAIN Site Configuration #

Typical deployment: Cologne data center

Configuration:

SITE_NAME=cologne
SITE_TYPE=main
MAIN_DB_HOST=localhost  # Local database
LOCAL_DB_HOST=localhost  # Same as main

Behavior:

All writes go directly to local database (no buffering)
All reads from local database
Redis used only for caching (not buffering)
Background processor disabled or minimal activity
LSN tracking not needed

Use cases:

Production main site
Development when testing direct operations
Any location with reliable, low-latency access to main database

SECONDARY Site Configuration #

Typical deployment: CCAT observatory, Chile

Configuration:

SITE_NAME=observatory
SITE_TYPE=secondary
MAIN_DB_HOST=main-db.example.com  # Remote main database
LOCAL_DB_HOST=localhost  # Local read-only replica

Behavior:

Critical writes buffered in Redis, executed asynchronously
Non-critical writes fail or redirect to main
Reads from local replica merged with buffered data
Redis used for buffering, read buffer, and caching
Background processor actively processes buffer
LSN tracking monitors replication state

Use cases:

Production observatory site
Development when testing transaction buffering
Any location with unreliable access to main database

Operation Routing #

Site Configuration Logic #

The should_buffer_operation() method determines buffering:

    def should_buffer_operation(self, operation_type: str) -> bool:
        """
        Determine if an operation should be buffered

        Args:
            operation_type: Type of operation ("critical", "non_critical")

        Returns:
            True if operation should be buffered
        """
        if not self.critical_operations_buffer:
            return False

        if operation_type == "critical":
            # Buffer critical operations on secondary sites
            return True

        if self.is_main_site:
            # Main site doesn't buffer operations
            return False

        # Don't buffer non-critical operations
        return False

Decision Tree #

        graph TD
    Start[Operation Request]
    CheckSite{Site Type?}
    CheckOp{Operation Type?}
    CheckBuffer{Buffering<br/>Enabled?}

    Start --> CheckSite
    CheckSite -->|MAIN| DirectWrite[Direct Write to DB]
    CheckSite -->|SECONDARY| CheckOp

    CheckOp -->|critical| CheckBuffer
    CheckOp -->|non-critical| DirectWrite

    CheckBuffer -->|true| Buffer[Buffer to Redis]
    CheckBuffer -->|false| DirectWrite

    Buffer --> Success[Return Immediately]
    DirectWrite --> Success

    style Buffer fill:#FFD700
    style DirectWrite fill:#90EE90

Database URL Selection #

The API selects appropriate database URLs based on site and operation:

def get_database_url(operation_type: str = "default") -> str:
    site_config = get_site_config()

    if site_config.is_main_site:
        # Main site: all operations use main database
        return site_config.get_main_database_url()
    else:
        # Secondary site: use local replica for reads
        # Background processor uses main for writes
        return site_config.get_local_database_url()

Redis Key Namespacing #

Redis keys are namespaced per site to prevent conflicts:

Key Prefix Structure #

# Base prefix
f"site:{site_name}"

# Transaction buffer
f"site:{site_name}:transaction_buffer"

# Failed transactions
f"site:{site_name}:failed_transactions"

# Transaction status
f"site:{site_name}:transaction:{transaction_id}"

# Cache
f"site:{site_name}:cache:{cache_key}"

Example Keys #

For site named “observatory”:

site:observatory:transaction_buffer
site:observatory:failed_transactions
site:observatory:transaction:abc-123-def
site:observatory:cache:observation_summary:456

This allows:

Multiple sites to share a Redis instance (testing)
Clear isolation between site data
Easy cleanup per site
Simple monitoring per site

Accessing Configuration #

In Code #

from ccat_ops_db_api.transaction_buffering import get_site_config

# Get global site configuration
site_config = get_site_config()

# Check site properties
if site_config.is_main_site:
    print("Running at main site")

if site_config.is_secondary_site:
    print("Running at secondary site")

# Get database URLs
main_url = site_config.get_main_database_url()
local_url = site_config.get_local_database_url()

# Get Redis keys
buffer_key = site_config.get_transaction_buffer_key()

# Check buffering decision
should_buffer = site_config.should_buffer_operation("critical")

Via API Endpoint #

curl http://localhost:8000/api/site/info

Response:

{
  "site_name": "observatory",
  "site_type": "secondary",
  "is_main_site": false,
  "is_secondary_site": true,
  "critical_operations_buffer": true,
  "transaction_buffer_size": 1000,
  "lsn_tracking_enabled": true
}

Configuration Best Practices #

Development Configuration #

Single database setup (simplest):

SITE_TYPE=main
MAIN_DB_HOST=localhost
LOCAL_DB_HOST=localhost
REDIS_HOST=localhost

Testing buffering (simulate observatory):

SITE_TYPE=secondary
MAIN_DB_HOST=localhost  # Pretend it's remote
LOCAL_DB_HOST=localhost  # Pretend it's replica
REDIS_HOST=localhost
CRITICAL_OPERATIONS_BUFFER=true

Production Configuration #

Main site (Cologne):

SITE_TYPE=main
MAIN_DB_HOST=localhost
LOCAL_DB_HOST=localhost
REDIS_HOST=redis.internal
CRITICAL_OPERATIONS_BUFFER=false

Secondary site (Observatory):

SITE_TYPE=secondary
MAIN_DB_HOST=db.cologne.example.com
LOCAL_DB_HOST=localhost
REDIS_HOST=localhost
CRITICAL_OPERATIONS_BUFFER=true
LSN_TRACKING_ENABLED=true

Security Considerations #

Sensitive values (never commit):

Database passwords
SECRET_KEY
GITHUB_CLIENT_SECRET
API tokens

Use environment variables or secrets management:

# Load from secrets manager
MAIN_DB_PASSWORD=$(aws secretsmanager get-secret-value --secret-id db-password --query SecretString --output text)

Configuration Validation #

Runtime Configuration Changes #

Not supported: Configuration changes require restart

Why:

Site type affects fundamental behavior
Connection pools established at startup
Background processor started based on site type

To change: Update configuration and restart the API

Monitoring Configuration #

Configuration Metrics #

Expose configuration via metrics:

# Prometheus-style metrics
site_type_info{site_name="observatory", site_type="secondary"} 1
buffering_enabled{site_name="observatory"} 1
lsn_tracking_enabled{site_name="observatory"} 1

Configuration Endpoint #

The /api/site/info endpoint provides real-time configuration state:

curl http://localhost:8000/api/site/info | jq .

Useful for:

Verifying deployment configuration
Debugging site-specific behavior
Monitoring dashboards
Integration tests

Summary #

Site configuration:

Determines behavior: MAIN (direct) vs SECONDARY (buffered)
Environment-driven: Environment variables > .env > settings.toml
Redis namespaced: Keys prefixed with site name
Validated at startup: Fails fast if misconfigured
Accessible at runtime: Via code and API endpoint

Key configuration parameters:

SITE_TYPE: “main” or “secondary”
MAIN_DB_HOST: Target for writes
LOCAL_DB_HOST: Target for reads
CRITICAL_OPERATIONS_BUFFER: Enable/disable buffering
LSN_TRACKING_ENABLED: Enable/disable replication tracking

Next Steps #

Authentication System - Authentication configuration
Transaction Buffering Overview - How buffering works
Running Locally with Docker Compose - Testing different configurations

Site Configuration#

This Page