Job Handler

jobhandling ¶

Thread-parallel job handling for posterior evaluations.

This module provides the interface for thread-parallel and asynchronous submission and retrieval of compute jobs for posterior evaluations. The functionality is based on Python's ThreadPoolExecutor.

Classes:

Name	Description
`JobHandler`	Thread-parallel job handling for model evaluations in the MLDA algorithm

mtmlda.core.jobhandling.JobHandler ¶

Handler for thread parallel job requests and retrieval.

Model evaluations correspond to the hierarchy of posteriors in the MLDA algorithm. The main components of the class are a Python ThreadPoolExecutor and a list of callables that resemble the model hierarchy. All jobs are associated with nodes in a Markov tree. The activity of these nodes is modified accordingly, along with the posterior of the node from the result of the model evaluation. The execution is non-blocking, but the precise nature of the callable dictates if the execution is actually parallel.

Methods:

Name	Description
`submit_job`	Submit a job to the threadpool executor
`get_finished_jobs`	Retrieve all jobs that have been fully executed
`some_job_is_done`	Check if any of the submitted jobs is done
`workers_available`	Check if free workers are available in the thread pool
`num_evaluations`	Returns number of jobs that have been submitted for each model

workers_available `property` ¶

workers_available: bool

Check if free workers are available in the thread pool.

num_evaluations `property` ¶

num_evaluations: list[int]

Return number of jobs that have been submitted for each model.

init ¶

__init__(
    executor: concurrent.ThreadPoolExecutor, models: Sequence[Callable], num_threads: int
) -> None

Constructor of the JobHandler.

The constructor only initializes data structures and does not perform any actions.

Parameters:

Name	Type	Description	Default
`executor`	`concurrent.ThreadPoolExecutor`	Python Threadpool object to use	required
`models`	`Sequence[Callable]`	List of callables that represent models to evaluate	required
`num_threads`	`int`	Number of threads in the pool	required

submit_job ¶

submit_job(node: at.AnyNode) -> None

Submit a job to the threadpool executor.

Jobs are associated with a node in the Markov tree. Upon submission, the node is marked as computing. Note that the computation is non-blocking, but the precise nature of the callable dictates if the execution is actually parallel.

Parameters:

Name	Type	Description	Default
`node`	`at.AnyNode`	Node for which to evaluate the model	required

get_finished_jobs ¶

get_finished_jobs() -> tuple[list[float], list[at.AnyNode]]

Retrieve all jobs that have been fully executed.

Submitted jobs in a ThreadPoolExecutor object are automatically checked for completion with the as_completed method. Nodes of completed jobs are marked as not computing. The routine assumes that the result of the submitted callable is a list of lists to match the UM-Bridge interface. Only the entry [0][0] is treated as the result of the computation.

Returns:

Type	Description
`tuple[list[float], list[at.AnyNode]]`	tuple[list[float], list[at.AnyNode]]: Lists of job results (model evaluation) and corresponding nodes

_some_job_is_done ¶

_some_job_is_done() -> bool

Check if any of the submitted jobs is done.