MPIRE allows you to provide shared objects to the workers in a similar way as is possible with the
multiprocessing.Process class. For the start method
fork these shared objects are treated as
which means they are only copied once changes are made to them. Otherwise they share the same memory address. This is
convenient if you want to let workers access a large dataset that wouldn’t fit in memory when copied multiple times.
The start method
fork isn’t available on Windows, which means copy-on-write isn’t supported there.
threading these shared objects are readable and writable without copies being made. For the start methods
forkserver the shared objects are copied once for each worker, in contrast to copying it for each
task which is done when using a regular
def task(dataset, x): # Do something with this copy-on-write dataset ... def main(): dataset = ... # Load big dataset with WorkerPool(n_jobs=4, shared_objects=dataset, start_method='fork') as pool: ... = pool.map(task, range(100))
Multiple objects can be provided by placing them, for example, in a tuple container.
Apart from sharing regular Python objects between workers, you can also share multiprocessing synchronization
primitives such as
multiprocessing.Lock using this method. Objects like these require to be shared through
inheritance, which is exactly how shared objects in MPIRE are passed on.
Shared objects are passed on as the second argument, after the worker ID (when enabled), to the provided function.
def main(): dataset = ... # Load big dataset with WorkerPool(n_jobs=4, start_method='fork') as pool: pool.set_shared_objects(dataset) ... = pool.map(task, range(100))
Shared objects have to be specified before the workers are started. Workers are started once the first
map call is
keep_alive=True and the workers are reused, changing the shared objects between two consecutive
map calls won’t work.
When copy-on-write is not available for you, you can also use shared objects to share a
multiprocessing.Value, or another object with
multiprocessing.Manager. You can then store results in the same
object from multiple processes. However, you should keep the amount of synchronization to a minimum when the resources
are protected with a lock, or disable locking if your situation allows it as is shown here:
from multiprocessing import Array def square_add_and_modulo_with_index(shared_objects, idx, x): # Unpack results containers square_results_container, add_results_container = shared_objects # Square, add and modulo square_results_container[idx] = x * x add_results_container[idx] = x + x return x % 2 def main(): # Use a shared array of size 100 and type float to store the results square_results_container = Array('f', 100, lock=False) add_results_container = Array('f', 100, lock=False) shared_objects = square_results_container, add_results_container with WorkerPool(n_jobs=4, shared_objects=shared_objects) as pool: # Square, add and modulo the results and store them in the results containers modulo_results = pool.map(square_add_and_modulo_with_index, enumerate(range(100)), iterable_len=100)
In the example above we create two results containers, one for squaring and for adding the given value, and disable locking for both. Additionally, we also return a value, even though we use shared objects for storing results. We can safely disable locking here as each task writes to a different index in the array, so no race conditions can occur. Disabling locking is, of course, a lot faster than having it enabled.