ADR 0001 — S3/LTA connection config lives on the DataLocation (self-serve)#

  • Status: Accepted

  • Date: 2026-06-19

  • Affects: ops-db (model + migration), ops-db-api, ops-db-ui, data-transfer

Context#

An S3 long-term-archive (LTA) location needs three things to be usable: a bucket/region (what it is), an endpoint URL (where to reach it), an access method (plain S3 via boto3, or Coscine), plus credentials.

Historically the S3DataLocation row stored only bucket_name and region. The endpoint URL was removed from the DB (migration a3e7c1f28d01) and resolved at runtime from data-transfer dynaconf settings (S3_ENDPOINTS, keyed by location name, with an S3_ENDPOINT_URL fallback). The access method was a single deployment-wide switch (S3_METHOD = boto3 | coscine) with global COSCINE_PROJECT / COSCINE_RESOURCE.

Consequences of that split:

  • Commissioning an S3 store was not self-serve. Adding a location in the UI did nothing useful until someone also edited data-transfer config in system-integration and redeployed — and the UI gave no hint of that coupling.

  • Silent fallback footgun. A location whose name had no S3_ENDPOINTS entry silently used the global default endpoint.

  • All-or-nothing method. Because S3_METHOD is global, every S3 location had to use the same method; you could not have one LTA on Coscine and another on plain S3 at the same time. Coscine mode also ignored the per-location bucket, collapsing every location onto one global Coscine resource.

The original argument for keeping the endpoint out of the DB was that it can be per-worker (different network vantage points reach the same bucket via different URLs, e.g. an in-cluster minio:9000 shortcut vs a public URL).

Decision#

Move the non-secret connection config onto S3DataLocation and make the admin UI the source of truth, under one operating rule that neutralises the per-worker objection:

Each S3 location has a single endpoint URL that is reachable from every worker that routes to it. Routing is single-site → its LTA, and we ensure the chosen endpoint is reachable from all those workers (i.e. we forgo in-cluster shortcut URLs in favour of one reachable URL).

New columns on s3_data_location (migration c7e2a9b41f08), all nullable:

Column

Meaning

endpoint_url

S3 endpoint (re-adds what a3e7c1f28d01 removed)

access_method

s3 (boto3) or coscine; NULL ⇒ global S3_METHOD

path_style

boto3 addressing: True path-style (RDS, MinIO), False virtual-host (NRW Datastorage), NULL ⇒ boto3 default / S3_PATH_STYLE

coscine_project

Coscine project id (identifier, not a secret)

coscine_resource

Coscine resource id (not a secret)

Secrets stay in the environment, keyed per-location as today: {SITE}_{LOCATION}_S3_ACCESS_KEY_ID / _SECRET_ACCESS_KEY, and now {SITE}_{LOCATION}_COSCINE_API_TOKEN, each with a global fallback.

Resolution is additive with fallback — data-transfer prefers the DB value and falls back to its existing dynaconf settings when the column is NULL:

endpoint = location.endpoint_url or S3_ENDPOINTS.get(name) or S3_ENDPOINT_URL
method   = location.access_method or S3_METHOD            # 's3'/'boto3' ≡ boto3 path
project  = location.coscine_project  or COSCINE_PROJECT
resource = location.coscine_resource or COSCINE_RESOURCE

Consequences#

  • Self-serve commissioning. An admin creates an S3/LTA location in the UI — bucket, region, endpoint, method (S3 or Coscine), and Coscine project/resource — and only drops the credentials/token into the deployment env once. No system-integration config edit or data-transfer redeploy.

  • Mixed backends. Method is per-location, so US can be plain S3 while Cologne is Coscine, simultaneously.

  • Non-breaking. Existing deployments keep working via the dynaconf fallback; locations migrate to DB-managed one at a time.

  • Trade-off accepted. A single endpoint_url per location means we give up per-worker-within-an-environment endpoints (e.g. an in-cluster minio:9000 shortcut alongside a public URL for the same bucket). Per-environment endpoints still work because each environment has its own database (dev’s row can hold minio:9000 while prod holds the public URL). This is acceptable under the single-reachable-endpoint operating rule above.

  • Secrets are never in the DB, so the DB is not a full connection descriptor by itself — credentials and the Coscine token remain deployment/env concerns.