asset_sync

Module for File Attachment synching

AssetSync

Class for managing AWS Deadline Cloud job-level attachments.

attachment_sync_inputs(s3_settings, attachments, queue_id, job_id, session_dir, fs_permission_settings=None, storage_profiles_path_mapping_rules={}, step_dependencies=None, on_downloading_files=None, os_env_vars=None)

Depending on the fileSystem in the Attachments this will perform two different behaviors: COPIED / None : downloads a manifest file and corresponding input files, if found. VIRTUAL: downloads a manifest file and mounts a Virtual File System at the specified asset root corresponding to the manifest contents

Parameters:

Name	Type	Description	Default
s3_settings	Optional[JobAttachmentS3Settings]	S3-specific Job Attachment settings.	required
attachments	Optional[Attachments]	an object that holds all input assets for the job.	required
queue_id	str	the ID of the queue.	required
job_id	str	the ID of the job.	required
session_dir	Path	the directory that the session is going to use.	required
fs_permission_settings	Optional[FileSystemPermissionSettings]	An instance defining group ownership and permission modes to be set on the downloaded (synchronized) input files and directories.	None
storage_profiles_path_mapping_rules	dict[str, str]	A dict of source path -> destination path mappings. If this dict is not empty, it means that the Storage Profile set in the job is different from the one configured in the Fleet performing the input-syncing.	{}
step_dependencies	Optional[list[str]]	the list of Step IDs whose output should be downloaded over the input job attachments.	None
on_downloading_files	Optional[Callable[[ProgressReportMetadata], bool]]	a function that will be called with a ProgressReportMetadata object for each file being downloaded. If the function returns False, the download will be cancelled. If it returns True, the download will continue.	None
os_env_vars	Dict[str, str] | None	environment variables to set for launched subprocesses	None

Returns:

Name	Type	Description
	SummaryStatistics	COPIED / None : a tuple of (1) final summary statistics for file downloads, and (2) a list of local roots for each asset root, used for path mapping.
VIRTUAL	List[Dict[str, str]]	same as COPIED, but the summary statistics will be empty since the download hasn't started yet.

copied_download(s3_settings, session_dir, fs_permission_settings=None, merged_manifests_by_root=dict(), on_downloading_files=None)

Parameters:

Name	Type	Description	Default
s3_settings	JobAttachmentS3Settings	S3-specific Job Attachment settings.	required
session_dir	Path	the directory that the session is going to use.	required
fs_permission_settings	Optional[FileSystemPermissionSettings]	An instance defining group ownership and permission modes to be set on the downloaded (synchronized) input files and directories.	None
merged_manifests_by_root	dict[str, BaseAssetManifest]	Merged manifests produced by _aggregate_asset_root_manifests()	dict()
on_downloading_files	Optional[Callable[[ProgressReportMetadata], bool]]	Callback when download files from S3.	None

Returns:

Type	Description
SummaryStatistics	The download summary statistics.

Raises:

Type	Description
JobAttachmentsS3ClientError	If any issue is encountered while downloading.

generate_dynamic_path_mapping(session_dir, attachments) staticmethod

Compute path mapping rules that are relative to the given session directory.

Parameters:

Name	Type	Description	Default
session_dir	Path	path to the current session directory	required
attachments	Attachments	an object that holds all input assets for the job.	required

Returns: a dictionary of local roots for each asset root, used for path mapping.

get_attachments(farm_id, queue_id, job_id)

Gets Job Attachment settings by calling the Deadline GetJob API.

get_local_destination(manifest_properties, dynamic_mapping_rules={}, storage_profiles_path_mapping_rules={}) staticmethod

Parameters:

Name	Type	Description	Default
manifest_properties	ManifestProperties	manifest properties to search local destination for.	required
dynamic_mapping_rules	dict[str, PathMappingRule]	manifest root path to worker host destination mapping relative to local session.	{}
storage_profiles_path_mapping_rules	dict[str, str]	a dict of source path -> destination path mappings.	{}

Returns: local destination corresponding to the given manifest properties. Raises: AssetSyncError If no path mapping rule is found for the given root path.

get_s3_settings(farm_id, queue_id)

Gets Job Attachment S3 settings by calling the Deadline GetQueue API.

sync_inputs(s3_settings, attachments, queue_id, job_id, session_dir, fs_permission_settings=None, storage_profiles_path_mapping_rules={}, step_dependencies=None, on_downloading_files=None, os_env_vars=None)

Depending on the fileSystem in the Attachments this will perform two different behaviors: COPIED / None : downloads a manifest file and corresponding input files, if found. VIRTUAL: downloads a manifest file and mounts a Virtual File System at the specified asset root corresponding to the manifest contents

Parameters:

Name	Type	Description	Default
s3_settings	Optional[JobAttachmentS3Settings]	S3-specific Job Attachment settings.	required
attachments	Optional[Attachments]	an object that holds all input assets for the job.	required
queue_id	str	the ID of the queue.	required
job_id	str	the ID of the job.	required
session_dir	Path	the directory that the session is going to use.	required
fs_permission_settings	Optional[FileSystemPermissionSettings]	An instance defining group ownership and permission modes to be set on the downloaded (synchronized) input files and directories.	None
storage_profiles_path_mapping_rules	dict[str, str]	A dict of source path -> destination path mappings. If this dict is not empty, it means that the Storage Profile set in the job is different from the one configured in the Fleet performing the input-syncing.	{}
step_dependencies	Optional[list[str]]	the list of Step IDs whose output should be downloaded over the input job attachments.	None
on_downloading_files	Optional[Callable[[ProgressReportMetadata], bool]]	a function that will be called with a ProgressReportMetadata object for each file being downloaded. If the function returns False, the download will be cancelled. If it returns True, the download will continue.	None
os_env_vars	Dict[str, str] | None	environment variables to set for launched subprocesses	None

Returns:

Name	Type	Description
	SummaryStatistics	COPIED / None : a tuple of (1) final summary statistics for file downloads, and (2) a list of local roots for each asset root, used for path mapping.
VIRTUAL	List[Dict[str, str]]	same as COPIED, but the summary statistics will be empty since the download hasn't started yet.

sync_outputs(s3_settings, attachments, queue_id, job_id, step_id, task_id, session_action_id, start_time, session_dir, storage_profiles_path_mapping_rules={}, on_uploading_files=None)

Uploads any output files specified in the manifest, if found.