# Archive Storage (S3 & Coscine)

```{eval-rst}
.. verified:: 2026-06-19
   :reviewer: Christof Buchbender
```

Long-term archive (LTA) data is written to object storage. Two access methods are
supported, selected per `S3DataLocation`:

- **boto3 S3** — any S3-compatible endpoint (NRW DataStore, NRW RDS-S3, MinIO)
- **Coscine** — the RWTH Coscine research-data platform (uses the Coscine SDK, not
  boto3)

Most connection details live **on the `S3DataLocation`** (set via the admin
infrastructure page — self-serve), so commissioning a new store needs no settings
deploy. Only **secrets** come from the environment. Each setting has a deployment-wide
fallback in `settings.toml` for when the location leaves it unset.

## Access method: Coscine vs S3

| Source | Field | Values |
|---|---|---|
| Per location (wins) | `S3DataLocation.access_method` | `"s3"` (⇒ boto3) or `"coscine"` |
| Deployment fallback | `S3_METHOD` | `"boto3"` or `"coscine"` |

Resolved by `utils.resolve_s3_method()`. `[staging]`/`[production]` default `S3_METHOD`
to `coscine`; a single location can still opt into boto3 by setting
`access_method="s3"`.

## Endpoint resolution

For boto3 locations the endpoint is resolved in `utils.get_s3_client()` in this order:

1. `S3DataLocation.endpoint_url` (set in the admin UI)
2. `S3_ENDPOINTS[location.name]` (per-env dict in `settings.toml`)
3. `S3_ENDPOINT_URL` (per-env global)

The endpoint **must be reachable from every worker that routes to this location** — auto
routing sends a site's data to its LTA, so the LTA endpoint has to be reachable from each
contributing site.

## Addressing style (the path-style knob)

S3 endpoints disagree on how the bucket is addressed, and using the wrong one silently
routes the request to the wrong place:

| Style | URL shape | Endpoints |
|---|---|---|
| path-style | `https://endpoint/bucket/key` | NRW **RDS-S3**, **MinIO** |
| virtual-host | `https://bucket.endpoint/key` | NRW **DataStore** |

Pointing **path-style** at the NRW DataStore lands the write on a read-only replication
site → `AccessDenied … site is not in the replication group and cannot write`. Switching
to **virtual-host** fixes it.

| Source | Field | Values |
|---|---|---|
| Per location (wins) | `S3DataLocation.path_style` | `True`=path, `False`=virtual-host, `NULL`=fallback |
| Deployment fallback | `S3_PATH_STYLE` | `true`=path, `false`=virtual-host, unset=boto3 "auto" |

Resolved by `utils._s3_boto_config()`. The `settings.toml` defaults:

| Environment | `S3_PATH_STYLE` | Why |
|---|---|---|
| `default` | `false` (virtual-host) | default endpoint is the NRW DataStore |
| `development`, `localdev` | `true` (path-style) | MinIO |
| `staging`, `production` | inherited `false` | inert for coscine; correct for a boto3/DataStore location commissioned there |

## Credentials (secrets — from the environment)

Secrets are **never stored in the database**. For boto3 locations,
`S3DataLocation.get_s3_credentials()` looks them up (via dynaconf) under:

```
{site.short_name}_{location.name}_S3_ACCESS_KEY_ID
{site.short_name}_{location.name}_S3_SECRET_ACCESS_KEY
```

Because data-transfer's dynaconf prefix is `CCAT_DATA_TRANSFER`, the actual environment
variables carry that prefix, e.g. for the Cologne site (`short_name="cologne"`) with a
`long_term_archive` location:

```
CCAT_DATA_TRANSFER_COLOGNE_LONG_TERM_ARCHIVE_S3_ACCESS_KEY_ID=...
CCAT_DATA_TRANSFER_COLOGNE_LONG_TERM_ARCHIVE_S3_SECRET_ACCESS_KEY=...
```

```{warning}
**Do not embed the site name in the location name.** The convention already prefixes
`site.short_name`, so a location literally named `cologne_long_term_archive` at site
`cologne` resolves to the doubled, easy-to-miss
`CCAT_DATA_TRANSFER_COLOGNE_COLOGNE_LONG_TERM_ARCHIVE_S3_ACCESS_KEY_ID`. Name the
location just `long_term_archive`.
```

If the per-location variable is unset (typo, missing prefix, or the doubling above), the
lookup **silently falls back** to the global `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY`
(default placeholder `"MUST_SET_VIA_ENV"`). The bogus key then surfaces at the S3 layer
as `InvalidAccessKeyId` — not as a config error.

Coscine locations use a token resolved the same way:
`{site.short_name}_{location.name}_COSCINE_API_TOKEN`, falling back to `COSCINE_API_TOKEN`.
The non-secret `coscine_project` / `coscine_resource` live on the location.

## Commissioning a new S3 LTA

1. **Admin UI** → infrastructure page → add an `S3` `DataLocation` on the site. Set
   `bucket_name`, `endpoint_url`, `region`, `access_method` (`s3` or `coscine`), and
   **addressing style** (Virtual-hosted for the DataStore, Path-style for RDS/MinIO).
2. **Secrets** → set the two env vars named exactly per the convention above (mind the
   `CCAT_DATA_TRANSFER_` prefix and the no-double-prefix rule), then restart the
   archive worker so dynaconf reloads.
3. **Verify** → trigger an archive and read the worker log line `S3 client configured`
   (see below).

## Troubleshooting

The worker logs the exact connection on every S3 client build
(`utils.get_s3_client()`):

- **INFO** `S3 client configured` — `endpoint`, `bucket`, `region`, `access_method`,
  `addressing_style`, masked `access_key`
- **DEBUG** `S3 credential fingerprint` — the `expected_env_var` it looked up plus
  first/last-char fingerprints of the access key and secret
- **WARNING** when the key is empty/`MUST_SET_VIA_ENV` — names the missing env var

| Error | Likely cause | Fix |
|---|---|---|
| `InvalidAccessKeyId` | per-location secret not resolved → placeholder sent (typo, missing prefix, doubled name) | check the `expected_env_var` in the logs; fix the env var name / location name |
| `AccessDenied … not in the replication group` | path-style against the NRW DataStore | set addressing to virtual-host (`path_style=false`) |
| `SignatureDoesNotMatch` | region or addressing disagrees with the endpoint (signing, not a wrong secret) | confirm `region` and addressing match the endpoint |