# ADR 0001 — S3/LTA connection config lives on the DataLocation (self-serve)

- Status: Accepted
- Date: 2026-06-19
- Affects: `ops-db` (model + migration), `ops-db-api`, `ops-db-ui`, `data-transfer`

## Context

An S3 long-term-archive (LTA) location needs three things to be usable: a
**bucket/region** (what it is), an **endpoint URL** (where to reach it), an
**access method** (plain S3 via boto3, or Coscine), plus **credentials**.

Historically the `S3DataLocation` row stored only `bucket_name` and `region`.
The endpoint URL was removed from the DB (migration `a3e7c1f28d01`) and resolved
at runtime from `data-transfer` dynaconf settings (`S3_ENDPOINTS`, keyed by
location name, with an `S3_ENDPOINT_URL` fallback). The access method was a
single deployment-wide switch (`S3_METHOD` = `boto3` | `coscine`) with global
`COSCINE_PROJECT` / `COSCINE_RESOURCE`.

Consequences of that split:

- **Commissioning an S3 store was not self-serve.** Adding a location in the UI
  did nothing useful until someone also edited `data-transfer` config in
  `system-integration` and redeployed — and the UI gave no hint of that coupling.
- **Silent fallback footgun.** A location whose `name` had no `S3_ENDPOINTS`
  entry silently used the global default endpoint.
- **All-or-nothing method.** Because `S3_METHOD` is global, every S3 location had
  to use the same method; you could not have one LTA on Coscine and another on
  plain S3 at the same time. Coscine mode also ignored the per-location bucket,
  collapsing every location onto one global Coscine resource.

The original argument for keeping the endpoint out of the DB was that it can be
**per-worker** (different network vantage points reach the same bucket via
different URLs, e.g. an in-cluster `minio:9000` shortcut vs a public URL).

## Decision

Move the **non-secret connection config onto `S3DataLocation`** and make the
admin UI the source of truth, under one operating rule that neutralises the
per-worker objection:

> **Each S3 location has a single endpoint URL that is reachable from every
> worker that routes to it.** Routing is single-site → its LTA, and we ensure
> the chosen endpoint is reachable from all those workers (i.e. we forgo
> in-cluster shortcut URLs in favour of one reachable URL).

New columns on `s3_data_location` (migration `c7e2a9b41f08`), all nullable:

| Column | Meaning |
|---|---|
| `endpoint_url` | S3 endpoint (re-adds what `a3e7c1f28d01` removed) |
| `access_method` | `s3` (boto3) or `coscine`; `NULL` ⇒ global `S3_METHOD` |
| `path_style` | boto3 addressing: `True` path-style (RDS, MinIO), `False` virtual-host (NRW Datastorage), `NULL` ⇒ boto3 default / `S3_PATH_STYLE` |
| `coscine_project` | Coscine project id (identifier, not a secret) |
| `coscine_resource` | Coscine resource id (not a secret) |

**Secrets stay in the environment**, keyed per-location as today:
`{SITE}_{LOCATION}_S3_ACCESS_KEY_ID` / `_SECRET_ACCESS_KEY`, and now
`{SITE}_{LOCATION}_COSCINE_API_TOKEN`, each with a global fallback.

**Resolution is additive with fallback** — `data-transfer` prefers the DB value
and falls back to its existing dynaconf settings when the column is `NULL`:

```
endpoint = location.endpoint_url or S3_ENDPOINTS.get(name) or S3_ENDPOINT_URL
method   = location.access_method or S3_METHOD            # 's3'/'boto3' ≡ boto3 path
project  = location.coscine_project  or COSCINE_PROJECT
resource = location.coscine_resource or COSCINE_RESOURCE
```

## Consequences

- **Self-serve commissioning.** An admin creates an S3/LTA location in the UI —
  bucket, region, endpoint, method (S3 or Coscine), and Coscine project/resource
  — and only drops the credentials/token into the deployment env once. No
  `system-integration` config edit or `data-transfer` redeploy.
- **Mixed backends.** Method is per-location, so US can be plain S3 while Cologne
  is Coscine, simultaneously.
- **Non-breaking.** Existing deployments keep working via the dynaconf fallback;
  locations migrate to DB-managed one at a time.
- **Trade-off accepted.** A single `endpoint_url` per location means we give up
  *per-worker-within-an-environment* endpoints (e.g. an in-cluster `minio:9000`
  shortcut alongside a public URL for the same bucket). Per-environment endpoints
  still work because each environment has its own database (dev's row can hold
  `minio:9000` while prod holds the public URL). This is acceptable under the
  single-reachable-endpoint operating rule above.
- Secrets are never in the DB, so the DB is not a full connection descriptor by
  itself — credentials and the Coscine token remain deployment/env concerns.