24. multi — Framework for multi-processing in pyFormex

This module contains some functions to perform multiprocessing inside pyFormex in a unified way.

24.1. Functions defined in module multi

multi.splitArgs(args, mask=None, nproc=-1, close=False)

Split data blocks over multiple processors.

Parameters:

• args: a list or tuple of data blocks. All items in the list that need to be split should be arrays with the same first dimension.
• mask: list of bool, with same length as args. It flags which items in the args list are to be split. If not specified, all array type items will be split.
• nproc: number of processors intended. If negative (default), it is set equal to the number of processors detected.
• close: bool. If True, the elements where the arrays are split are included in both blocks delimited by the element.

Returns a list of nproc tuples. Each tuple contains the same number of items as the input args and in the same order, whereby the (nonmasked) arrays are replaced by a slice of the array along its first axis, and the masked and non-array items are replicated as is.

This function uses arraytools.splitar() for the splitting of the arrays.

Example:

>>> splitArgs([np.arange(5),'abcde'],nproc=3)
[(array([0, 1]), 'abcde'), (array([2]), 'abcde'), (array([3, 4]), 'abcde')]
>>> for i in splitArgs([np.eye(5),'=>',np.arange(5)],nproc=3):
...     print("%s %s %s" % i)
[[ 1. 0. 0. 0. 0.]
 [ 0. 1. 0. 0. 0.]] => [0 1]
[[ 0. 0. 1. 0. 0.]] => [2]
[[ 0. 0. 0. 1. 0.]
 [ 0. 0. 0. 0. 1.]] => [3 4]
>>> for i in splitArgs([np.eye(5),'=>',np.arange(5)],mask=[1,0,0],nproc=3):
...     print("%s %s %s" % i)
[[ 1. 0. 0. 0. 0.]
 [ 0. 1. 0. 0. 0.]] => [0 1 2 3 4]
[[ 0. 0. 1. 0. 0.]] => [0 1 2 3 4]
[[ 0. 0. 0. 1. 0.]
 [ 0. 0. 0. 0. 1.]] => [0 1 2 3 4]

multi.dofunc(arg)

Helper function for the multitask function.

It expects a tuple with (function,args) as single argument.

multi.multitask(tasks, nproc=-1)

Perform tasks in parallel.

Runs a number of tasks in parallel over a number of subprocesses.

Parameters:

• tasks : a list of (function,args) tuples, where function is a callable and args is a tuple with the arguments to be passed to the function.
• ` nproc`: the number of subprocesses to be started. This may be different from the number of tasks to run: processes finishing a task will pick up a next one. There is no benefit in starting more processes than the number of tasks or the number of processing units available. The default will set nproc to the minimum of these two values.

multi.worker(input, output)

Helper function for the multitask function.

This is the function executed by any of the processes started by the multitask function. It takes tuples (function,args) from the input queue, computes the results of the call function(args), and pushes these results on the output queue.

Parameters:

• input: Queue holding the tasks to be performed
• output: Queue where the results are to be delivered

multi.multitask2(tasks, nproc=-1)

Perform tasks in parallel.

Runs a number of tasks in parallel over a number of subprocesses.

Parameters:

• tasks : a list of (function,args) tuples, where function is a callable and args is a tuple with the arguments to be passed to the function.
• ` nproc`: the number of subprocesses to be started. This may be different from the number of tasks to run: processes finishing a task will pick up a next one. There is no benefit in starting more processes than the number of tasks or the number of processing units available. The default will set nproc to the minimum of these two values.