Archive Storage (S3 & Coscine)#

✓
Documentation Verified Last checked: 2026-06-19 Reviewer: Christof Buchbender

Long-term archive (LTA) data is written to object storage. Two access methods are supported, selected per S3DataLocation:

  • boto3 S3 — any S3-compatible endpoint (NRW DataStore, NRW RDS-S3, MinIO)

  • Coscine — the RWTH Coscine research-data platform (uses the Coscine SDK, not boto3)

Most connection details live on the S3DataLocation (set via the admin infrastructure page — self-serve), so commissioning a new store needs no settings deploy. Only secrets come from the environment. Each setting has a deployment-wide fallback in settings.toml for when the location leaves it unset.

Access method: Coscine vs S3#

Source

Field

Values

Per location (wins)

S3DataLocation.access_method

"s3" (⇒ boto3) or "coscine"

Deployment fallback

S3_METHOD

"boto3" or "coscine"

Resolved by utils.resolve_s3_method(). [staging]/[production] default S3_METHOD to coscine; a single location can still opt into boto3 by setting access_method="s3".

Endpoint resolution#

For boto3 locations the endpoint is resolved in utils.get_s3_client() in this order:

  1. S3DataLocation.endpoint_url (set in the admin UI)

  2. S3_ENDPOINTS[location.name] (per-env dict in settings.toml)

  3. S3_ENDPOINT_URL (per-env global)

The endpoint must be reachable from every worker that routes to this location — auto routing sends a site’s data to its LTA, so the LTA endpoint has to be reachable from each contributing site.

Addressing style (the path-style knob)#

S3 endpoints disagree on how the bucket is addressed, and using the wrong one silently routes the request to the wrong place:

Style

URL shape

Endpoints

path-style

https://endpoint/bucket/key

NRW RDS-S3, MinIO

virtual-host

https://bucket.endpoint/key

NRW DataStore

Pointing path-style at the NRW DataStore lands the write on a read-only replication site → AccessDenied … site is not in the replication group and cannot write. Switching to virtual-host fixes it.

Source

Field

Values

Per location (wins)

S3DataLocation.path_style

True=path, False=virtual-host, NULL=fallback

Deployment fallback

S3_PATH_STYLE

true=path, false=virtual-host, unset=boto3 “auto”

Resolved by utils._s3_boto_config(). The settings.toml defaults:

Environment

S3_PATH_STYLE

Why

default

false (virtual-host)

default endpoint is the NRW DataStore

development, localdev

true (path-style)

MinIO

staging, production

inherited false

inert for coscine; correct for a boto3/DataStore location commissioned there

Credentials (secrets — from the environment)#

Secrets are never stored in the database. For boto3 locations, S3DataLocation.get_s3_credentials() looks them up (via dynaconf) under:

{site.short_name}_{location.name}_S3_ACCESS_KEY_ID
{site.short_name}_{location.name}_S3_SECRET_ACCESS_KEY

Because data-transfer’s dynaconf prefix is CCAT_DATA_TRANSFER, the actual environment variables carry that prefix, e.g. for the Cologne site (short_name="cologne") with a long_term_archive location:

CCAT_DATA_TRANSFER_COLOGNE_LONG_TERM_ARCHIVE_S3_ACCESS_KEY_ID=...
CCAT_DATA_TRANSFER_COLOGNE_LONG_TERM_ARCHIVE_S3_SECRET_ACCESS_KEY=...

Warning

Do not embed the site name in the location name. The convention already prefixes site.short_name, so a location literally named cologne_long_term_archive at site cologne resolves to the doubled, easy-to-miss CCAT_DATA_TRANSFER_COLOGNE_COLOGNE_LONG_TERM_ARCHIVE_S3_ACCESS_KEY_ID. Name the location just long_term_archive.

If the per-location variable is unset (typo, missing prefix, or the doubling above), the lookup silently falls back to the global S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY (default placeholder "MUST_SET_VIA_ENV"). The bogus key then surfaces at the S3 layer as InvalidAccessKeyId — not as a config error.

Coscine locations use a token resolved the same way: {site.short_name}_{location.name}_COSCINE_API_TOKEN, falling back to COSCINE_API_TOKEN. The non-secret coscine_project / coscine_resource live on the location.

Commissioning a new S3 LTA#

  1. Admin UI → infrastructure page → add an S3 DataLocation on the site. Set bucket_name, endpoint_url, region, access_method (s3 or coscine), and addressing style (Virtual-hosted for the DataStore, Path-style for RDS/MinIO).

  2. Secrets → set the two env vars named exactly per the convention above (mind the CCAT_DATA_TRANSFER_ prefix and the no-double-prefix rule), then restart the archive worker so dynaconf reloads.

  3. Verify → trigger an archive and read the worker log line S3 client configured (see below).

Troubleshooting#

The worker logs the exact connection on every S3 client build (utils.get_s3_client()):

  • INFO S3 client configured — endpoint, bucket, region, access_method, addressing_style, masked access_key

  • DEBUG S3 credential fingerprint — the expected_env_var it looked up plus first/last-char fingerprints of the access key and secret

  • WARNING when the key is empty/MUST_SET_VIA_ENV — names the missing env var

Error

Likely cause

Fix

InvalidAccessKeyId

per-location secret not resolved → placeholder sent (typo, missing prefix, doubled name)

check the expected_env_var in the logs; fix the env var name / location name

AccessDenied … not in the replication group

path-style against the NRW DataStore

set addressing to virtual-host (path_style=false)

SignatureDoesNotMatch

region or addressing disagrees with the endpoint (signing, not a wrong secret)

confirm region and addressing match the endpoint