Core Utilities#
Fundamental components including database connections, configuration, exceptions, and utility functions.
Core Components:
- Database
- Configuration
- Exceptions
CCATDataOperationErrorScheduleErrorNetworkErrorStorageErrorDataCorruptionErrorArchiveErrorPackageErrorDeletionErrorPermanentErrorServiceUnavailableErrorServiceExitNoSecondaryRoutesErrorDatabaseOperationErrorDataTransferErrorBBCPErrorDestinationFileExistsErrorSegmentationFaultErrorConfigurationErrorDatabaseErrorRetryableErrorNonRetryableErrorArchiveCreationErrorUnpackErrorChecksumVerificationErrorArchiveCorruptionError- Overview
- Key Components
- Related Documentation
- Utilities
get_redis_connection()get_s3_client()service_shutdown()unique_id()create_archive()unpack_local()unpack_remote()calculate_checksum()make_bbcp_command()create_local_folder()create_remote_folder()make_long_term_archive_copy_command()run_ssh_command()check_remote_folder_size_gb()parse_bbcp_output()calculate_transfer_rate()generate_readable_filename()get_s3_key_for_package()get_s3_key_for_file()- Overview
- Key Components
- Related Documentation
- Decorators
- Logging Utils
- Metrics
- Notification Service
- Operation Types
- Queue Discovery
- Settings Manager
Overview#
Core utilities provide the foundation for the data transfer system:
- Database
Database connection and ORM models.
- Configuration
System configuration management and settings.
- Exceptions
Custom exception classes for error handling.
- Utilities
Common utility functions and helpers.
- Decorators
Reusable decorators for common patterns.
- Logging Utils
Logging configuration and utilities.
- Metrics
Performance metrics and monitoring.
- Notification Service
System notifications and alerts.
- Operation Types
Data models for operation types and states.
- Queue Discovery
Celery queue discovery and management.
- Settings Manager
Dynamic settings management.
Core Architecture#
These components provide the foundation that all managers and services build upon:
Database connectivity and data models
Configuration management
Error handling and logging
Performance monitoring
System notifications
For understanding how these components fit into the larger system:
Core Concepts - Core concepts and data model
Philosophy & Design Principles - System design principles