22. multi — Framework for multi-processing in pyFormex

This module contains some functions to perform multiprocessing in pyFormex in a unified way.

22.1. Functions defined in module multi

multi.splitArgs(args, mask=None, nproc=-1, close=False)[source]

Split data blocks over multiple processors.

Parameters:
  • args (list or tuple of array_like) – A sequence of data blocks that may need to be split for parallel processing over multiple processors. Splitting is done along the first dimension, which should therefore be the same for all arrays in the sequence that need to be split.
  • mask (list or tuple of bool) – If provided, this flags the items in args that should be split. The list should have the same length as args. If not provided, all array type items in args will be split.
  • nproc (int) – Intended number of processors. If negative (default), it is set equal to the number of processors detected on the host machine.
  • close (bool) – If True, the elements where the arrays are split are included in both blocks delimited by the element. Thus splitting an array [1, 2, 3] in two results in [1, 2] and [2, 3], while the default split would be [1, 2] and [3].
Returns:

list of tuples. – The list contains nproc tuples and each tuple contains the same number of items as the input args and in the same order, whereby the (nonmasked) arrays are replaced by a slice of the array along its first axis, and the masked and non-array items are replicated as is.

See also

arraytools.splitar()
the low level function used to do the splitting

Examples

>>> splitArgs([np.arange(5),'abcde'],nproc=3)
[(array([0, 1]), 'abcde'), (array([2]), 'abcde'), (array([3, 4]), 'abcde')]
>>> for i in splitArgs([np.eye(5),'=>',np.arange(5)],nproc=3):
...     print("%s %s %s" % i)
[[ 1. 0. 0. 0. 0.]
 [ 0. 1. 0. 0. 0.]] => [0 1]
[[ 0. 0. 1. 0. 0.]] => [2]
[[ 0. 0. 0. 1. 0.]
 [ 0. 0. 0. 0. 1.]] => [3 4]
>>> for i in splitArgs([np.eye(5),'=>',np.arange(5)],mask=[1,0,0],nproc=3):
...     print("%s %s %s" % i)
[[ 1. 0. 0. 0. 0.]
 [ 0. 1. 0. 0. 0.]] => [0 1 2 3 4]
[[ 0. 0. 1. 0. 0.]] => [0 1 2 3 4]
[[ 0. 0. 0. 1. 0.]
 [ 0. 0. 0. 0. 1.]] => [0 1 2 3 4]
multi.dofunc(arg)[source]

Helper function for the multitask function.

Parameters:arg (tuple) – The first item of the tuple is a callable. The remaining items are its arguments.
Returns:object – The value of the callable when passed the remaining items as arguments.

Examples

>>> dofunc((max,(2, 5, 3)))
5
multi.multitask(tasks, nproc=-1)[source]

Perform tasks in parallel.

Runs a number of tasks in parallel over a number of subprocesses.

Parameters:
  • tasks (list of tuples) – Each task in the list is a tuple where the first item is a callable and the other items are the arguments to be passed to the callable.
  • nproc (int) – The number of subprocesses to be started. This may be different from the number of tasks to run: processes finishing a task will pick up a next one. There is no benefit in starting more processes than the number of tasks or the number of processing units available. The default will set nproc to the minimum of these two values.

Examples

>>> task1 = (int.__add__, (2, 3))
>>> task2 = (int.__mul__, (2, 3))
>>> multitask((task1, task2))
[5, 6]