Data Flow Examples#

Complete data flow examples showing request-to-response patterns for different scenarios.

UI Read Request#

Simple query flow without buffering:

        sequenceDiagram
    participant UI as Web Frontend
    participant API as FastAPI
    participant Auth as Authentication
    participant Router
    participant DB as Local Database

    UI->>API: GET /api/transfer/overview
    API->>Auth: Verify JWT
    Auth-->>API: User authenticated
    API->>Router: Route to handler
    Router->>DB: Query transfers
    DB-->>Router: Transfer data
    Router-->>API: Format response
    API-->>UI: JSON response (< 100ms)

Observatory Write (Buffered)#

Critical operation with transaction buffering:

        sequenceDiagram
    participant Script as Observatory Script
    participant API
    participant Builder as Transaction Builder
    participant Manager as Transaction Manager
    participant Redis
    participant BG as Background<br/>Processor
    participant MainDB as Main DB
    participant LSN as LSN Tracker
    participant Replica

    Script->>API: POST /executed_obs_units/start
    API->>Builder: Build transaction
    Builder->>Builder: Generate UUID
    Builder-->>API: Transaction
    API->>Manager: Buffer transaction
    Manager->>Redis: LPUSH to buffer
    Manager->>Redis: Cache generated ID
    Redis-->>Manager: OK
    Manager-->>API: Transaction ID
    API-->>Script: 201 Created (< 20ms)

    Note over Script: Client continues immediately

    loop Background Processing
        BG->>Redis: RPOP from buffer
        BG->>MainDB: Execute transaction
        MainDB-->>BG: Success
        BG->>MainDB: Capture LSN
        MainDB-->>BG: LSN: 0/12345678
        BG->>LSN: Track replication
        LSN->>Replica: Poll replay LSN
        alt Replicated
            Replica-->>LSN: LSN caught up
            LSN-->>BG: Replicated
            BG->>Redis: Cleanup caches
        else Not yet
            Replica-->>LSN: Still behind
            LSN-->>BG: Not replicated
            BG->>Redis: Extend cache TTL
        end
    end

Observatory Read (Smart Query)#

Query merging database + buffer:

        sequenceDiagram
    participant Script
    participant API
    participant Smart as Smart Query<br/>Manager
    participant DB as Local Replica
    participant Redis as Redis Buffer
    participant ReadBuf as Read Buffer

    Script->>API: GET /executed_obs_units/123
    API->>Smart: search_records(...)

    par Parallel Queries
        Smart->>DB: Query database
        Smart->>Redis: Query buffered cache
        Smart->>ReadBuf: Query read buffer
    end

    DB-->>Smart: [obs1, obs2]
    Redis-->>Smart: [obs3 (buffered)]
    ReadBuf-->>Smart: [updates to obs3]

    Smart->>Smart: Merge (buffer > DB)
    Smart->>Smart: Apply read buffer updates
    Smart->>Smart: Deduplicate by ID

    Smart-->>API: [obs1, obs2, obs3 (merged)]
    API-->>Script: JSON response

WebSocket Real-Time Updates#

Streaming updates via WebSocket:

        sequenceDiagram
    participant UI
    participant API
    participant Redis as Redis<br/>Pub/Sub
    participant DB

    UI->>API: Connect WS /api/transfer/ws/overview
    API->>API: Authenticate token
    API->>Redis: Subscribe to "transfer_updates"
    API->>DB: Query initial data
    DB-->>API: Current overview
    API->>UI: Send initial data

    Note over UI,DB: Real-time updates

    loop On Data Changes
        DB->>Redis: PUBLISH transfer_updates
        Redis-->>API: Update message
        API->>UI: Send update
    end

Complete Observatory Operation#

End-to-end flow from start to finish:

  1. Start observation (buffered)

  2. Query observation (smart query, shows buffered)

  3. Finish observation (update read buffer)

  4. Query again (smart query, shows updated)

  5. Background processing executes buffered start

  6. Background processing executes buffered finish

  7. LSN tracking confirms replication

  8. Cache cleanup occurs

  9. Future queries read from database (normal)

Summary#

Key patterns:

  • UI reads: Direct database queries (fast, simple)

  • Operations writes: Buffered (reliable, never block)

  • Operations reads: Smart queries (merge buffer + DB)

  • Real-time: WebSockets with Redis pub/sub

Next Steps#