Celery broker SSL via config dicts, not URL query#

ops-db-api #126, epic #129. Relates to ADR-0001 (live-check contract) and the #121 broker-URL escape-hatch.

Context#

POST /api/admin/diagnose/{type}/{id}/live-check crashes in staging/prod while the producer (ccat_ops_db_api/live_check/producer.py) initialises the Celery result backend:

ValueError: A rediss:// URL must have parameter ssl_cert_reqs and this must be
set to CERT_REQUIRED, CERT_OPTIONAL, or CERT_NONE

The previous design packed SSL into the broker URL query (rediss://…?ssl_cert_reqs=…&ssl_ca_certs=…). That single URL string is consumed by three libraries, each parsing ssl_cert_reqs with an incompatible accepted spelling:

  • kombu (the Celery broker transport),

  • Celery’s RedisBackend (the result backend), and

  • redis-py via Redis.from_url in get_broker_redis() (the correlation client).

The pinned versions are the trap, and the two failure modes are distinct exceptions raised at different points:

  • Celery’s RedisBackend → ValueError at backend init. A bare rediss:// result-backend URL with no ssl_cert_reqs (in the URL or a redis_backend_use_ssl dict) is rejected when the backend is constructed — app.backend parses the URL without connecting. This is the exact ValueError in the prod traceback above, and it surfaces at the very first live-check request, before any Redis socket is opened. Celery 5.5.3 itself accepts both the CERT_* and lowercase ssl_cert_reqs forms.

  • redis-py 8.0.0 → RedisError at connect time. redis-py accepts only the lowercase required/none spelling; an ssl_cert_reqs=CERT_REQUIRED carried in the URL is not rejected at Redis.from_url(...) construction but raises redis.exceptions.RedisError (“Invalid SSL Certificate Requirements Flag: CERT_REQUIRED”) only when a connection is instantiated from the pool. Note RedisError is not a subclass of ValueError — it is a different exception type surfacing at a different time.

There is therefore no single ssl_cert_reqs spelling in the URL that is safe for all three consumers: the CERT_REQUIRED form Celery is happy with is exactly the form redis-py rejects, while the lowercase form redis-py needs is fine for Celery but must reach it via the config dict, not the bare-URL backend that RedisBackend refuses outright.

Decision#

Stop putting SSL in the URL. Mirror data-transfer’s proven production pattern (ccat_data_transfer/setup_celery_app.py): a bare rediss://[:pw@]host:port/db URL plus SSL supplied through Celery’s dedicated config dicts and redis-py kwargs.

  • build_broker_url() returns a bare URL. When celery_broker_url is injected (the #121 escape-hatch) its host/password/db are kept but any ssl_* query is stripped (_strip_ssl_query).

  • A shared _broker_ssl_options() -> dict | None derives {ssl_cert_reqs, ssl_ca_certs, ssl_certfile, ssl_keyfile} from celery_broker_ssl_verify_cert (→ "required" / "none") and the existing redis_ca_cert/redis_certfile/redis_keyfile layout. It returns None when SSL is disabled (non-rediss injected URL, or celery_broker_ssl_enabled false). Keys match the data-transfer spelling exactly.

  • get_producer_app() sets conf.broker_use_ssl and conf.redis_backend_use_ssl from that one dict (only when not None).

  • get_broker_redis() builds the redis-py client with SSL via kwargs (ssl=True + the ssl_* keys), never from the URL query.

The lowercase "required"/"none" spelling is chosen because it is the only one redis-py 8.x accepts, and Celery 5.5 also accepts it — so one dict feeds all three consumers.

This supersedes the #121 URL escape-hatch for SSL params only. Host, password and db still come from the injected URL; SSL no longer does.

Considered options#

  • (a) Bare URL + config dicts (chosen). SSL leaves the URL entirely; the one spelling that satisfies all three parsers is set programmatically. Matches the data-transfer worker the producer talks to.

  • (b) Pick one URL spelling. Impossible: no spelling satisfies both redis-py 8.x (required/none only) and the CERT_*-shaped deploy strings already in the field.

  • (c) Per-consumer URL rewriting. Rewrite the query differently for kombu vs RedisBackend vs redis-py. Rejected: fragile, three code paths, and Celery owns the broker/backend URL parsing internally — there is no clean seam to inject per-consumer query strings.

Consequences#

  • Deploy ordering (image before compose). The new image tolerates an injected full ?ssl_* URL (it strips the query), so it is safe to roll the image out before the compose/inject change that drops ssl_* from the URL. The reverse is not required: a bare-URL inject also works with the old image only if that image still reads SSL from settings — so the documented safe order is image first, then compose.

  • The producer no longer reads SSL from the URL query at all; deployments that still inject ?ssl_* are silently tolerated (query stripped) but the params are ignored — SSL comes from celery_broker_ssl_* + the cert settings.

  • Injected-URL scheme is authoritative for the SSL decision. When a celery_broker_url is injected, _broker_ssl_options() keys the on/off decision off that URL’s scheme, not celery_broker_ssl_enabled. So injecting a plain redis:// URL while celery_broker_ssl_enabled=true turns SSL silently off (the injected URL wins). This is intended and safe — an operator injecting a bare redis:// is explicitly asking for a plaintext broker — but is non-obvious, so set the scheme to match the intent.

  • Dev/local is unchanged: SSL off → plain redis://, no use_ssl dicts, no SSL kwargs.

  • The boundary holds only as long as no future code re-introduces ssl_* into any broker URL. New broker wiring must route SSL through _broker_ssl_options().