Data Flow Examples#

Complete data flow examples showing request-to-response patterns for different scenarios.

Transfer Overview Read (Operation Lists)#

The transfer overview (GET /api/transfer/overview) does not read a per-stage status column off each artifact. It reads the uniform Operation chain off the package and derives everything — the funnel, the failed list, the per-kind counts — from the operation rows. The model itself is described once in the ops-db Operation model reference; the identity-vs-routing split is in the data-transfer two-axes page. The read side here only navigates that model; it does not redefine it.

        sequenceDiagram
    participant UI as Web Frontend
    participant API as FastAPI
    participant Auth as Authentication
    participant Router
    participant Crud as crud.get_transfer_overview
    participant DB as Local Database

    UI->>API: GET /api/transfer/overview
    API->>Auth: Verify JWT (admin actions only; overview is anonymous)
    API->>Router: Route to handler
    Router->>Crud: build overview
    Crud->>DB: COUNT operations per OperationKind
    Crud->>DB: Load non-archived RawDataPackages
    Crud->>DB: _attach_operations — bulk-load 5 op lists BY FK
    DB-->>Crud: packages + their operation chains
    Crud->>Crud: classify_transferring_step (seven-step funnel)
    Crud->>Crud: attribute failures latest-step-first
    Crud-->>Router: TransferOverview (no error text)
    Router-->>API: Format response
    API-->>UI: JSON response (< 100ms)

How the read side reads the package#

The overview builder (crud.get_transfer_overview → build_package_funnel) walks each package’s operation chain rather than any paired status column:

  • _attach_operations bulk-loads five operation lists by FK. There is no artifact→operation ORM relationship — operations carry forward FKs only — so the read side resolves each stage with a fixed number of bulk queries (one per kind: packaging, bundling, transfer, unpack, archive) keyed on raw_data_package_id / data_transfer_package_id. This is what avoids an N+1: the queries are independent of the number of packages, attaching packaging_operations, bundling_operations, transfer_operations, unpack_operations, and archive_operations as plain lists on each package.

  • classify_transferring_step places each in-flight package in the seven-step funnel. It walks the first incomplete operation in pipeline order transfer → unpack → awaiting_archive → archiving. The occupancy invariant is that every package lands in exactly one cell: Σ(occupancy + failed) over all seven steps plus the inconsistent overflow equals the total package count. The awaiting_archive / archiving split distinguishes the at-rest archive backlog (transfer + unpack done, no archive op yet) from the in-flight archive set (an archive op exists), per ADR-0003.

  • Failure attribution is charged latest-step-first to the failing Operation. attribute_failed_package scans the chain from the end backwards (archiving → unpack → transfer → bundling → packaging) and charges the failure to the first currently-FAILED operation. The breadcrumb it emits is keyed by the operation’s OperationKind.value plus the Operation row id — the same key the durable failure-history/{operation_type}/{operation_id} endpoint uses. Error text is read server-side only to pick the funnel cell; it is never shipped in the anonymous overview payload (ADR-0002). A state=FAILED package with no currently-failed op is reconciled to the explicit “inconsistent” bucket (ADR-0004) rather than silently dropped.

Observatory Write (Buffered)#

Critical operation with transaction buffering:

        sequenceDiagram
    participant Script as Observatory Script
    participant API
    participant Builder as Transaction Builder
    participant Manager as Transaction Manager
    participant Redis
    participant BG as Background<br/>Processor
    participant MainDB as Main DB
    participant LSN as LSN Tracker
    participant Replica

    Script->>API: POST /executed_obs_units/start
    API->>Builder: Build transaction
    Builder->>Builder: Generate UUID
    Builder-->>API: Transaction
    API->>Manager: Buffer transaction
    Manager->>Redis: LPUSH to buffer
    Manager->>Redis: Cache generated ID
    Redis-->>Manager: OK
    Manager-->>API: Transaction ID
    API-->>Script: 201 Created (< 20ms)

    Note over Script: Client continues immediately

    loop Background Processing
        BG->>Redis: RPOP from buffer
        BG->>MainDB: Execute transaction
        MainDB-->>BG: Success
        BG->>MainDB: Capture LSN
        MainDB-->>BG: LSN: 0/12345678
        BG->>LSN: Track replication
        LSN->>Replica: Poll replay LSN
        alt Replicated
            Replica-->>LSN: LSN caught up
            LSN-->>BG: Replicated
            BG->>Redis: Cleanup caches
        else Not yet
            Replica-->>LSN: Still behind
            LSN-->>BG: Not replicated
            BG->>Redis: Extend cache TTL
        end
    end

Observatory Read (Smart Query)#

Query merging database + buffer:

        sequenceDiagram
    participant Script
    participant API
    participant Smart as Smart Query<br/>Manager
    participant DB as Local Replica
    participant Redis as Redis Buffer
    participant ReadBuf as Read Buffer

    Script->>API: GET /executed_obs_units/123
    API->>Smart: search_records(...)

    par Parallel Queries
        Smart->>DB: Query database
        Smart->>Redis: Query buffered cache
        Smart->>ReadBuf: Query read buffer
    end

    DB-->>Smart: [obs1, obs2]
    Redis-->>Smart: [obs3 (buffered)]
    ReadBuf-->>Smart: [updates to obs3]

    Smart->>Smart: Merge (buffer > DB)
    Smart->>Smart: Apply read buffer updates
    Smart->>Smart: Deduplicate by ID

    Smart-->>API: [obs1, obs2, obs3 (merged)]
    API-->>Script: JSON response

WebSocket Real-Time Updates#

Streaming updates via WebSocket:

        sequenceDiagram
    participant UI
    participant API
    participant Redis as Redis<br/>Pub/Sub
    participant DB

    UI->>API: Connect WS /api/transfer/ws/overview
    API->>API: Authenticate token
    API->>Redis: Subscribe to "transfer_updates"
    API->>DB: Query initial data
    DB-->>API: Current overview
    API->>UI: Send initial data

    Note over UI,DB: Real-time updates

    loop On Data Changes
        DB->>Redis: PUBLISH transfer_updates
        Redis-->>API: Update message
        API->>UI: Send update
    end

Complete Observatory Operation#

End-to-end flow from start to finish:

  1. Start observation (buffered)

  2. Query observation (smart query, shows buffered)

  3. Finish observation (update read buffer)

  4. Query again (smart query, shows updated)

  5. Background processing executes buffered start

  6. Background processing executes buffered finish

  7. LSN tracking confirms replication

  8. Cache cleanup occurs

  9. Future queries read from database (normal)

Summary#

Key patterns:

  • Transfer overview reads: Navigate the uniform Operation chain — the funnel, failed list, and per-kind counts are derived from operation rows (_attach_operations by FK, classify_transferring_step), not from a paired per-stage status column on the artifact.

  • Operations writes: Buffered (reliable, never block)

  • Operations reads: Smart queries (merge buffer + DB)

  • Real-time: WebSockets with Redis pub/sub

Next Steps#