We can improve performace of python calculation by running python in parallel. In this turtorial we will be making use of the multithreading library to run python code in parallel.
Multiprocessing is part of the standard python library distribution on versions python/2.6 and above so no additonal instalation is required (Owens and Pitzer both offer 2.7 and above so this should not be an issue). However, we do recommend you use python environments when using multiple libraries to avoid version conflicts with different projects you may have. See here for more information.
Please note that this parallelization is limited to a single node. If you need to run your job across multiple nodes, you should consider other options like mpi4py.
Pool
One way to parallelizing is by created a parallel pool. This can be done by using the Pool method:
p = Pool(10)
This will create a pool of 10 worker processes.
Once you have a pool of worker processes created you can then use the map method to assign tasks to each worker.
p.map(my_function, something_iterable)
Here is an example python code:
from multiprocessing import Pool
from timeit import default_timer as timer
import time
def sleep_func(x):
time.sleep(x)
if __name__ == '__main__':
arr = [1,1,1,1,1]
# create a pool of 5 worker processes
p = Pool(5)
start = timer()
# assign sleep_func to a worker for each entry in arr.
# each array entry is passed as an argument to sleep_func
p.map(sleep_func, arr)
print("parallel time: ", timer() - start)
start = timer()
# run the functions again but in serial
for a in arr:
sleep_func(a)
print("serial time: ", timer() - start)
The above code was then submitted using the below job script:
#!/bin/bash #SBATCH --account <your-project-id> #SBATCH --job-name Python_ExampleJob #SBATCH --nodes=1 #SBATCH --time=00:10:00 module load python python example_pool.py
After submitting the above job, the following was the output:
parallel time: 1.003282466903329 serial time: 5.005984931252897
See the documenation for more details and examples on using Pool.
Process
The mutiprocessing library also provides the Process method to run functions asynchronously.
To create a Process object you can simply make a call to:
proc = Process(target=my_function, args=[my_function, arguments, go, here])
The target is set equal to the name of your function which you want to run asynchronously and args is a list of arguement for your function.
Start running a process asynchronously by:
proc.start()
Doing so will begin running the function in another process and the main parent process will continue in its execution.
You can make the parent process wait for a child process to finish with:
proc.join()
If you use proc.run() it will run your process and wait for it to finish before continuing on in executing the parent process.
Note: The below code will start proc2 only after proc1 has finshed. If you want to start multiple processes and wait for them use start() and join() instead of run.
proc1.run() proc2.run()
Examples
Here some example code:
from multiprocessing import Process
from timeit import default_timer as timer
import time
def sleep_func(x):
print(f'Sleeping for {x} sec')
time.sleep(x)
if __name__ == '__main__':
# initialize process objects
proc1 = Process(target=sleep_func, args=[1])
proc2 = Process(target=sleep_func, args=[1])
# begin timer
start = timer()
# start processes
proc1.start()
proc2.start()
# wait for both process to finish
proc1.join()
proc2.join()
print('Time: ', timer() - start)
Running this code give the following output:
Sleeping for 1 sec Sleeping for 1 sec Time: 1.0275288447737694
You can create a many process easily in loop aswell:
from multiprocessing import Process
from timeit import default_timer as timer
import time
def sleep_func(x):
print(f'Sleeping for {x} sec')
time.sleep(x)
if __name__ == '__main__':
# empty list to later store processes
processes = []
# start timer
start = timer()
for i in range(10):
# initialize and start processes
p = Process(target=sleep_func, args=[5])
p.start()
# add the processes to list for later reference
processes.append(p)
# wait for processes to finish.
# we cannot join() them within the same loop above because it would
# wait for the process to finish before looping and creating the next one.
# So it would be the same as running them sequentially.
for p in processes:
p.join()
print('Time: ', timer() - start)
Output:
Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Sleeping for 5 sec Time: 5.069192241877317
See documentation for more information and example on using Process.
Shared States
When running process in parallel it is generally best to avoid sharing states between processes. However, if data must be shared see documentation for more information and examples on how to safely share data.
Other Resources
- Spark:You can also drastically improve preformance of your python code by using Apache Spark. See Spark for more details.
- Horovod: If you are using Tensorflow, PyTorch or other python machine learning packages you may want to also consider using Horovod. Horovod will take single-GPU training scripts and scale it to train across many GPUs in parallel.
