progress_tracker
DownloadSummaryStatistics
dataclass
¶
Bases: SummaryStatistics
A summary statistics metadata to be returned to the client when the downloading files has completed. In addition to the general statistics, includes a dict mapping download locations to the number of downloaded files in each of those locations.
ProgressReportMetadata
dataclass
¶
A metadata (with defined key-value pairs) about the progress to be reported back to client during file upload/downloads. Within this metadata will be a status message and progress(%) of the hashing, uploads or downloads of files.
ProgressStatus
¶
Bases: Enum
Represents the current stage of asset/file processing
DOWNLOAD_IN_PROGRESS = ('DOWNLOAD_IN_PROGRESS', 'Downloaded')
class-attribute
instance-attribute
¶
Downloading files
NONE = ('NONE', '')
class-attribute
instance-attribute
¶
The asset manager is not assigned any work.
PREPARING_IN_PROGRESS = ('PREPARING_IN_PROGRESS', 'Processed')
class-attribute
instance-attribute
¶
The asset manager is hashing files.
SNAPSHOT_IN_PROGRESS = ('SNAPSHOT_IN_PROGRESS', 'Snapshotted')
class-attribute
instance-attribute
¶
Snapshotting files
UPLOAD_IN_PROGRESS = ('UPLOAD_IN_PROGRESS', 'Uploaded')
class-attribute
instance-attribute
¶
The asset manager is uploading files.
ProgressTracker
dataclass
¶
A class that records the progress of file processing, and reports the progress data back to the client using callbacks passed from the client. The process is one of the following - hashing, uploading, or downloading.
__init__(status, total_files, total_bytes, on_progress_callback=None, callback_interval=CALLBACK_INTERVAL, max_files_in_chunk=MAX_FILES_IN_CHUNK, logger=None, log_interval=LOG_INTERVAL, log_percentage_threshold=LOG_PERCENTAGE_THRESHOLD)
¶
get_download_summary_statistics(downloaded_files_paths_by_root)
¶
Returns the summary statistics of download operation.
get_summary_statistics()
¶
Returns the summary statistics of hashing or upload operation.
increase_processed(num_files=1, file_bytes=0)
¶
Adds the number and size of processed files.
increase_skipped(num_files=1, file_bytes=0)
¶
Adds the number and size of skipped files.
set_total_files(total_files, total_bytes)
¶
Stores the number and size of files to be processed.
SummaryStatistics
dataclass
¶
A summary statistics metadata to be returned to the client when processing of files
(hashing or uploading) has completed.
The skipped_files refers to:
- if this statistics is for hashing operation: the number of files whose hashing is
skipped by the hash cache.
- if this statistics is for uploading operation: the number of files that have already
been uploaded to S3 bucket and thus skipped uploading.
aggregate(other)
¶
Aggregates other object of SummaryStatistics to this.