Managers

Core manager classes that orchestrate data transfer operations. These classes follow the Manager/Worker pattern described in Philosophy & Design Principles.

Manager Classes:

• Raw Data Package Manager
  • RawDataPackageOperations
  • create_raw_data_package_task
  • is_same_host()
  • execute_remote_command()
  • get_unpackaged_raw_data_files()
  • group_files_by_execution_and_module()
  • get_primary_buffer_for_site()
  • create_raw_data_packages_for_location()
  • get_sites_with_source_locations()
  • create_raw_data_packages()
  • raw_data_package_manager_service()
  • Overview
  • Key Functions
  • Related Documentation
• Data Transfer Package Manager
  • DataTransferPackageOperations
  • create_data_transfer_package_task
  • get_unpackaged_raw_data_packages_in_buffers()
  • group_packages_for_transfer()
  • discover_automatic_routes()
  • discover_secondary_routes()
  • find_route_overrides()
  • get_primary_buffer_for_site()
  • get_next_lta_site_round_robin()
  • create_primary_data_transfers()
  • create_secondary_data_transfers()
  • create_data_transfer_packages_for_buffer()
  • schedule_data_transfer_package_creation()
  • create_data_transfer_packages()
  • data_transfer_package_manager_service()
  • create_primary_data_transfer_packages()
  • get_sites_with_buffer_locations()
  • validate_site_configuration()
  • get_transfer_statistics()
  • Overview
  • Key Functions
  • Related Documentation
• Transfer Manager
  • DataTransferTask
  • transfer_files_bbcp
  • transfer_transfer_packages()
  • Overview
  • Key Functions
  • Related Documentation
• Archive Manager
  • Overview
  • Key Functions
  • Related Documentation
• Deletion Manager
  • Overview
  • Key Functions
  • Related Documentation
• Staging Manager
  • StagingTask
  • stage_data_task
  • process_staging_jobs()
  • get_processing_locations_for_site()
  • get_sites_with_processing_locations()
  • Overview
  • Key Functions
  • Related Documentation

Overview

Managers are responsible for specific stages of the data pipeline:

Raw Data Package Manager

Creates packages from raw instrument data files.

Data Transfer Package Manager

Manages packages during site-to-site transfers.

Transfer Manager

Orchestrates actual data transfers between locations.

Archive Manager

Handles long-term archival to LTA systems.

Deletion Manager

Manages cleanup and deletion policies.

Staging Manager

Handles data staging for processing.

Manager/Worker Pattern

Each manager follows the same architectural pattern:

1. Manager Process: - Scans database for work to be done - Prepares operations (creates DB records, validates) - Submits Celery tasks to appropriate queues - Runs anywhere with database access

2. Worker Process: - Listen to queues for their location/operation type - Perform actual work (copy files, verify checksums, etc.) - Update database with results - Must run on machines with data access

See Philosophy & Design Principles for detailed explanation of this pattern.