Parallelize independent computation across processors
CPU-bound calculations
Python · Module
multiprocessing
multiprocessing provides process-based concurrency through an API resembling threading. It can use multiple processors and includes Pool for distributing a function across input values.
How it works
The module starts separate processes and provides process-oriented concurrency that can use multiple processors without being limited by the Global Interpreter Lock. Its Pool abstraction distributes a function over input values, while Process supports individually managed child processes.
Syntax
multiprocessing.Process(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)
multiprocessing.Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, context=None)Parameters
| Name | Type | Description |
|---|---|---|
target / func | callable | Callable executed by a child process or pool worker. |
args / iterable | tuple / iterable | Arguments supplied to the target callable or values passed to a pool operation. |
processes | int | None | Number of worker processes created by a pool; if omitted, the pool chooses a default. |
Returns
The module provides process, pool, communication, synchronization, and shared-state objects; individual return values depend on the API used.
Key points
- It offers a process-based API whose shape resembles threading, making familiar lifecycle concepts available for separate processes.
- Separate worker processes can execute work across multiple processors, which suits CPU-intensive workloads.
- The module is available on the two major platform families named by the documentation: POSIX systems and Windows.
- Pool groups reusable worker processes so callers can submit tasks without creating a new process for every input.
- A pool can apply one function across a collection of values, providing a data-parallel programming model.
- Beyond threading-like operations, the module includes process-specific capabilities that have no direct threading equivalent.
Common mistakes
- Protect process-starting code with an
if __name__ == "__main__":guard, especially for portable code that must work with process start methods on different platforms. - Processes have separate memory, so state is not automatically shared as it is between threads; use the module's inter-process communication or shared-memory facilities when needed.
Example: Map work across processes
from multiprocessing import Pool
def square(value):
return value * value
if __name__ == "__main__":
with Pool() as pool:
print(pool.map(square, [1, 2, 3, 4]))Creates a worker-process pool and applies a top-level function to multiple inputs.
When to use it
Apply one function to many inputs with a worker pool
Batch transformation
Run independently managed child processes
Isolated service tasks
Exchange data between processes with queues, pipes, or managers
Cross-process coordination
Interview questions
QWhy use multiprocessing instead of threading for CPU-bound work?
multiprocessing uses subprocesses rather than threads, allowing Python programs to use multiple processors while avoiding the Global Interpreter Lock. It supports both local and remote concurrency.
Follow-up: Why can process-based parallelism require explicit communication? Each process has its own execution context, so coordination or data exchange must use multiprocessing communication or shared-state mechanisms.
QWhat problem does Pool solve?
Pool represents a group of worker processes and can distribute a function over multiple input values. pool.map() returns the mapped results in input order after the work completes.
Follow-up: When might a manually managed Process be preferable? Use it when the application needs independent process lifecycles or control that does not fit a pool's task-distribution model.
QHow do you start and wait for a process?
A child process is created by constructing a Process object with a target callable and starting it. The parent can wait for completion with the process object's join operation.
Follow-up: What portability concern should be handled around process creation? Put process-launching code under the main-module guard so a spawned child does not execute the parent’s startup code during import.
Related
multiprocessing.Processmultiprocessing.Poolmultiprocessing.Queuemultiprocessing.Pipemultiprocessing.shared_memorymultiprocessing.Managermultiprocessing.get_context