Multiprocessing in Python

Last Updated on March 21, 2023 by Prepbytes

Multiprocessing is used for dividing the tasks into smaller parts that can be performed to accomplish the tasks in a lesser amount of time, however, multiple systems that can be a combination of multiple processors, systems, specialized hardware, etc. need to be a part of the execution involved in multiprocessing at the same interval of time.

What is Multiprocessing?

Multiprocessing can be simply termed in terms of a task where multiple processors of a computer work together by splitting that single task and distributing it among them to give better efficiency. We can take up an example of a sous chef cooking all the dishes, if the sous chef gets a bunch of assistant chefs to help him cook different meals then the task can be done faster and more efficiently.

To put more clarity on multiprocessing in python, a process in a program is executed exclusively from all the other processes. Each process gets its own share of resources on execution, like memory and CPU. Also, the process can communicate with the other processes using inter-process communication.

With the rise in the number of cores in processors in modern-day systems, multiprocessing becomes more desirable to use for better computational powers. Below is the given illustration with multiple processes divided into a number of threads.

There are different techniques and libraries to perform multiprocessing in different programming languages. In the python multiprocessing focused article, let us learn how we can implement multiprocessing.

Multiprocessing in Python

In python, multiprocessing can be implemented using the multiprocessing module to spawn processes for them to execute in parallel. Thus, the python multiprocessing module helps to optimize tasks and save time.

Python Multiprocessing module constitute of three main classes that are Process, Queue, and Lock to design and maintain a parallel program. It provides an API for creating or managing processes, inter-process communication and synchronization.

• Creation of Process
  In order to create a new process, you must define a function that will be executing the process and then pass the function as a target while constructing the process object. Here is the code to it:-

  Code:

  • Python

  import multiprocessing
  
  def my_function():
      print("Hello from process", multiprocessing.current_process().name)
  
  p = multiprocessing.Process(target=my_function)
  p.start()
  p.join()
  
  

  Explanation:
  In this example, we define a function my_function that simply prints a message. We then create a new Process object and specify the target argument as the my_function function. Finally, we start the new process with p.start() and wait for it to finish with p.join(). Explanation

• Communication between Processes
  Processes are isolated from each other which indicates that they are not able to share data with each other. However, the module named multiprocessing in python has pipes, queues and shared memory that enable inter-process communication.

  Pipes allow interprocess communication by initiating a connection between the processes where one process writes to the pipe and other process reads from the pipe.

  Code:

  • Python

  import multiprocessing
  
  def send_message(conn):
      conn.send("Hello from the sender process!")
  
  def receive_message(conn):
      print(conn.recv())
  
  conn1, conn2 = multiprocessing.Pipe()
  
  p1 = multiprocessing.Process(target=send_message, args=(conn1,))
  p2 = multiprocessing.Process(target=receive_message,args=(conn2,))
  
  p1.start()
  p2.start()
  
  p1.join()
  p2.join()
  
  

  Explanation:
  In this example, we create two processes, send_message and receive_message, and a pipe connection between them. The sender process sends a message through the pipe with conn.send(), and the receiver process reads the message with conn.recv().

• Queue
  Another mechanism made for multiprocessing in python where multiple processes are allowed to enqueue items into a queue and retrieve or deque items from the queue in a First In First Out manner.

  Code:

  • Python

  import multiprocessing
  
  def produce_items(queue):
      for i in range(5):
          queue.put(i)
      queue.put(None) 
  
  def consume_items(queue):
      while True:
          item = queue.get()
          if item is None:
              break
          print("Consumed", item)
  
  # Create a multiprocessing queue
  queue = multiprocessing.Queue()
  
  p1 = multiprocessing.Process(target=produce_items, args=(queue,))
  p2 = multiprocessing.Process(target=consume_items, args=(queue,))
  
  p1.start()
  p2.start()
  
  p1.join()
  p2.join()
  
  

  Explanation:
  In this example, we create two processes, produce_items() for producing and consume_items() for consuming, and a queue object between them. The producer process adds five items to the queue with queue.put(), and then adds a None item to signal the end of the queue. The consumer process retrieves item from the queue using queue.get() and prints it.

Applications of Multiprocessing in Python

Now that we have some strong ideas related to python multiprocessing, let us look at the most common use cases for python multiprocessing that is making our lives easier.

1. Development of GUI
   Multiprocessing is used to design and develop a graphical user interface such that GUI is responsive even when multiple processes are running in the background.

2. Distributed Computing
   A Task can be distributed among multiple machines connected over a network and it comes in handy when a large amount of computing power is required e.g big data analysis or deep learning.

3. Multithreading
   It is built on top of the concept of multiprocessing, here the different threads are at work to perform different tasks simultaneously.

4. Miscellaneous
   Multiprocessing is used in tasks such as matrix multiplication, image processing, simulation, web scraping, I/O operations, data processing etc.

Conclusion
In this article, we studied what is multiprocessing, how it differs from multithreading and read about the python multiprocessing library and how the Pipe, Queue and Process objects can be used to achieve various tasks. Hope you liked this article and we expect to see you again at PrepBytes with another informative article.

Frequently Asked Questions

1. What is multiprocessing in Python?
Multiprocessing is a module in Python that enables the execution of multiple processes in parallel on a computer with multiple processors or cores. It is useful for tasks that require heavy computational processing, such as matrix multiplication, image processing, and simulations.

2. How does multiprocessing differ from multithreading?
Multiprocessing and multithreading are two ways of achieving parallelism in Python. Multithreading involves creating multiple threads within a single process, while multiprocessing involves creating multiple processes that run concurrently. In general, multiprocessing is better for CPU-bound tasks, while multithreading is better for I/O-bound tasks.

3. How do you create a new process in multiprocessing?
To create a new process in multiprocessing, you need to create a Process object and pass in a target function to be executed in the new process. You can also pass in any arguments to the target function using the args parameter.

4. How do you share data between processes in multiprocessing?
To share data between processes in multiprocessing, you can use shared memory objects such as Value and Array, or use a queue object such as Queue.

5. How do you handle errors in multiprocessing?
To handle errors in multiprocessing, you can use the try-except block to catch any exceptions that occur in the child process. You can also use the Queue object to send error messages from the child process to the parent process. It is important to handle errors properly in multiprocessing to prevent the entire program from crashing.