- resqpy.multiprocessing.function_multiprocessing(function: Callable, kwargs_list: List[Dict[str, Any]], recombined_epc: Union[Path, str], cluster, consolidate: bool = True, require_success=False, tmp_dir_path: Union[Path, str] = '.', backend: str = 'dask') List[bool] ¶
Calls a function concurrently with the specfied arguments.
A multiprocessing pool is used to call the function multiple times in parallel. Once all results are returned, they are combined into a single epc file.
function (Callable) –
the wrapper function to be called. Needs to return:
index (int): the index of the kwargs in the kwargs_list.
- success (bool): whether the function call was successful, whatever that
epc_file (Path/str): the epc file path where the objects are stored.
uuid_list (List[str]): list of UUIDs of relevant objects.
kwargs_list (List[Dict[Any]]) – A list of keyword argument dictionaries that are used when calling the function.
recombined_epc (Path/str) – A pathlib Path or path string of where the combined epc will be saved.
cluster – if using the Dask backend, a LocalCluster is a Dask cluster on a local machine. If using a job queing system, a JobQueueCluster can be used such as an SGECluster, SLURMCluster, PBSCluster, LSFCluster etc.
consolidate (bool) – if True and an equivalent part already exists in a model, it is not duplicated and the uuids are noted as equivalent.
require_success (bool) – if True and any instance fails, then an exception is raised
tmp_dir_path (str) – path where the temporary directory is saved. Defaults to the calling code directory.
backend (str) – the joblib parallel backend used. Dask is used by default so a Dask cluster must be passed to the cluster argument.
success_list (List[bool]) – A boolean list of successful function calls.
This function uses the Dask backend by default to run the given function in parallel, so a Dask cluster must be setup and passed as an argument if Dask is used. Dask will need to be installed in the Python environment because it is not a dependency of the project. More info can be found at https://resqpy.readthedocs.io/en/latest/tutorial/multiprocessing.html