Logging Utils#

ccat_data_transfer.logging_utils.set_log_correlation(**fields: Any)[source]#

Set the correlation fields auto-injected into structured log lines.

Returns the ContextVar token so the caller can restore the previous context with reset_log_correlation() once the task body returns.

ccat_data_transfer.logging_utils.reset_log_correlation(token) → None[source]#

Restore the correlation context to the state before set_log_correlation.

ccat_data_transfer.logging_utils.get_log_correlation() → dict[source]#

Return the currently-active correlation fields (empty outside a task).

class ccat_data_transfer.logging_utils.BBCPLogHandler(base_log_path: str = '/var/log/ccat/bbcp_transfers')[source]#

Bases: object

__init__(base_log_path: str = '/var/log/ccat/bbcp_transfers')[source]#
get_log_path(transfer_id: int, timestamp: datetime | None = None) → Path[source]#

Generate structured log path for BBCP transfer.

store_bbcp_output(session: Session, data_transfer: DataTransfer, stdout: bytes, stderr: bytes, success: bool, timestamp: datetime | None = None) → DataTransferLog[source]#

Store BBCP output and create minimal log entry.

class ccat_data_transfer.logging_utils.LogfmtFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)[source]#

Bases: Formatter

Formats log records as a single logfmt line.

Produces: ts=<utc-iso8601> level=<lowercase> logger=<short-module> <record.getMessage()>

Tracebacks are appended on following lines (no leading ts= prefix) so that downstream multiline collectors that key on ‘^ts=’ fold them into the event.

format(record: LogRecord) → str[source]#

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

ccat_data_transfer.logging_utils.configure_logging(level: Any | None = None) → None[source]#

Install exactly one LogfmtFormatter handler on the package base logger.

Idempotent: if a LogfmtFormatter handler is already present on the base logger, only the level is updated. This function is safe to call multiple times (issue #124 will call it each poll cycle to change level at runtime).

Parameters:

level – logging level name (“INFO”, “DEBUG”, …) or int constant. Defaults to settings.LOG_LEVEL (“INFO” unless overridden).

ccat_data_transfer.logging_utils.apply_runtime_log_level(op_config, session) → None[source]#

Read LOG_LEVEL from operational config and apply it. Never raises.

class ccat_data_transfer.logging_utils.StructuredLogger(logger: Logger)[source]#

Bases: object

Wrapper for structured logging with consistent logfmt formatting.

Each call to debug/info/warning/error/exception formats the message body as msg="<text>" key=value ... and delegates to the underlying stdlib logger, which carries the body through to LogfmtFormatter where the ts/level/logger prefix fields are prepended once.

__init__(logger: Logger)[source]#
setLevel(level: int) → None[source]#

Set the logging level of the logger.

getEffectiveLevel() → int[source]#

Get the effective logging level of the logger.

debug(message: str, **kwargs) → None[source]#

Log a debug message.

info(message: str, **kwargs) → None[source]#

Log an info message.

warning(message: str, **kwargs) → None[source]#

Log a warning message.

error(message: str, error: Exception | None = None, **kwargs) → None[source]#

Log an error message.

If error is a BaseException, summary fields (error_type, error) go on the main logfmt line and the traceback is carried via exc_info so LogfmtFormatter appends it on following lines — keeping the summary single-line. If error is a plain string (legacy callers), it is embedded directly as the error= field with no traceback.

critical(message: str, **kwargs) → None[source]#

Log a critical message.

exception(message: str, **kwargs) → None[source]#

Log an exception with traceback at ERROR level. Must be called from an except block — logger.exception sets exc_info=True itself, so LogfmtFormatter appends the traceback on following lines.

ccat_data_transfer.logging_utils.get_structured_logger(name: str) → StructuredLogger[source]#

Return a StructuredLogger for the given module name.

Ensures the package base logger (‘ccat_data_transfer’) has a LogfmtFormatter handler installed via configure_logging() so output always reaches stderr. Module-level loggers propagate to the base logger – no per-logger handler needed.

ccat_data_transfer.logging_utils.setup_celery_logging()[source]#

Configure Celery logging to use the same format as our application logging

Overview#

Logging configuration and utilities for the data transfer system.

Key Components#

  • Logging configuration

  • Log formatting and handlers

  • Logging utilities and helpers