Deep Dive#

This section provides detailed technical documentation on the core systems and implementations within ops-db-api.

Overview#

The deep dive documentation covers:

  • Transaction Buffering: Complete buffering system architecture

  • Authentication: Unified auth implementation details

  • Routers: Endpoint organization and implementation

  • Data Flow: Complete request/response patterns

  • Caching Strategy: Redis caching patterns and policies

These guides are intended for developers who need to:

  • Understand implementation details

  • Extend the system with new features

  • Debug complex issues

  • Optimize performance

Deep Dive Sections#

Transaction Buffering#

The transaction buffering system is the core innovation that enables reliable operations despite network issues.

Topics covered:

  • System lifecycle and architecture

  • Transaction builder and dependencies

  • Redis buffering and queue management

  • Background processing patterns

  • LSN-based replication tracking

  • Smart query merging

  • Read buffer for mutable updates

See Transaction Buffering for details.

Authentication#

Unified authentication supporting both interactive users and automation.

Topics covered:

  • Unified authentication interface

  • GitHub OAuth flow implementation

  • API token generation and management

  • Role-based access control

  • Permission system

See Authentication Deep Dive for details.

Routers#

Endpoint organization, categorization, and implementation patterns.

Topics covered:

  • UI-focused routers

  • Operations-focused routers

  • Shared routers

  • Request/response patterns

See Routers Deep Dive for details.

Data Flow#

Complete data flow examples from request to response.

See Data Flow Examples for examples.

Caching Strategy#

Redis caching patterns, TTL strategies, and invalidation.

See Caching Strategy for details.

Prerequisites#

Before diving into these topics, you should be familiar with:

  • FastAPI framework basics

  • SQLAlchemy ORM

  • Redis data structures

  • PostgreSQL replication concepts

  • Async Python (asyncio)

Recommended reading order:

  1. Start with philosophy: Design Philosophy - Understand the “why”

  2. Then architecture: Architecture Overview - Understand the “what”

  3. Then deep dives: This section - Understand the “how”

  4. Finally tutorials: Tutorials - Apply the knowledge

Code References#

All deep dive documentation includes extensive code references using literalinclude directives that point to actual implementation files.

Key source directories:

  • ccat_ops_db_api/transaction_buffering/ - Buffering system

  • ccat_ops_db_api/auth/ - Authentication

  • ccat_ops_db_api/routers/ - API endpoints

  • ccat_ops_db_api/dependencies.py - Dependency injection

  • ccat_ops_db_api/main.py - Application setup

Conventions#

Code Examples#

Code examples are shown in two ways:

  1. Inline examples - Simplified, pedagogical code

  2. literalinclude - Actual production code with line references

# Inline example (simplified)
def simple_example():
    return "hello"

import logging
import os
from contextlib import asynccontextmanager
from typing import Dict, Set

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

from .routers import (
    auth,

Diagrams#

Mermaid diagrams for:

  • Sequence diagrams (request flows)

  • Architecture diagrams (system components)

  • State machines (transaction lifecycle)

ASCII diagrams for:

  • Simple flows

  • Tree structures

Cross-References#

Extensive cross-references connect related topics:

  • :doc: - Links to other documentation pages

  • :ref: - Links to specific sections

  • :meth: - Links to methods (when using Sphinx autodoc)

Testing Deep Dive Concepts#

Each deep dive topic includes suggestions for hands-on testing:

Transaction Buffering:

# Set as secondary site
export SITE_TYPE=secondary

# Monitor buffer
watch -n 1 curl -s http://localhost:8000/buffer-stats

Authentication:

# Test JWT
curl -H "Authorization: Bearer $JWT_TOKEN" http://localhost:8000/auth/me

# Test API token
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8000/auth/me

Caching:

# Monitor Redis
redis-cli MONITOR

Debugging Tips#

Enable verbose logging:

export LOG_LEVEL=DEBUG
uvicorn ccat_ops_db_api.main:app --reload --log-level debug

Use API health endpoints:

curl http://localhost:8000/health
curl http://localhost:8000/buffer-stats
curl http://localhost:8000/api/site/info

Inspect Redis directly:

redis-cli
> LLEN site:observatory:transaction_buffer
> LRANGE site:observatory:transaction_buffer 0 -1

Next Steps#

Choose a deep dive topic: