Looking through some of the posts here, it seems that people could benefit from a modern Python multi-threading tutorial.
For those who don't know, threading is a way to have your program do multiple things at once. Take for example you are trying to process a massive text file. You need to run a function that processes each line of the text file individually. You could write a program that looks like the following:
In a lot of cases this is probably good enough, but as that text file becomes very large your program will become slower and slower because of the need to process every line individually and wait for the line to finish before moving on to the next one.
This is where threading comes in. Threading will allow you to process X number of lines at a time in parallel. This means that you can process a whole lot more data much quicker.
Fortunately for us, Python 3 provides some very useful libraries for handling Threads and making sure that everything starts and stops as intended.
Since threads are spun up and run asynchronously we will need to add a bit of extra code that will allow us to control the number of threads active at a given time, and also monitor all of our active threads and wait for them all to finish before allowing our program to close.
To do this we will use something called a "Thread Pool".
To help you to understand what a thread pool is a why it is useful, I will use the following analogy.
Image a busy bar at peek hours. There is a huge line out the door of people who are trying to get in to get a drink. There are only 5 or 6 bartenders on staff tonight and it was agreed before hand that there can only be a max of 50 people at the bar at a time otherwise the bartenders will become too overwhelmed to server everyone. The bouncer's job outside is to control the flow of people into the bar. He constantly monitors the number of people at the bar and only allows another person in to be served when the number of people at the bar is less than 50.
This helps to illustrate the problem that we face with multi threading. If we spin up an unlimited amount of threads, then we won't have the resources to serve all the function calls. This will result in errors and information not being processed correctly. Just to draw a direct comparison, the bar tenders are analogous to the function being run on each line. The customers are the number of threads and the bouncer is the pool manager. He is managing a pool of 50 people (threads) and never letting there be any more than 50 people (active threads) be served at any one time.
Now that you know what a thread pool is and why it is important, let's rewrite the above code using a thread pool.
As you can see, we didn't have to change much. All that really changed is how we are now calling doStuff()
We import a class called ThreadPoolExecutor that will handle the thread pool for us. Then instead of calling doStuff() directly we pass the function and the arguments to a method called "submit()" in the executor object. This adds a new thread to the pool that will be executed when < 5 threads are active.
The final for loop allows us to check for when a thread is completed. We iterate over the existing threads in the pool (thread_pool) and check if it is done using the .done() method. While the thread has not completed we wait. When the while condition is no longer satisfied we print the results and print the results of that thread.
And that's it! This is an incredibly useful too to have in your tool belt. It took me a while to fully understand but once I did I was able to use it everywhere. Some common issues people run into is synchronization of information and excess resource consumption. Managing threads and determine why they failed and when is also a common challenge. That is all for another post though.
I hope this helped! If you have any more questions feel free to ask. I am happy to help.
For those who don't know, threading is a way to have your program do multiple things at once. Take for example you are trying to process a massive text file. You need to run a function that processes each line of the text file individually. You could write a program that looks like the following:
Code
def doStuff(line):
# Do stuff to line
return line
with open("big_test_file.txt") as btf:
for line in btf:
doStuff(line)In a lot of cases this is probably good enough, but as that text file becomes very large your program will become slower and slower because of the need to process every line individually and wait for the line to finish before moving on to the next one.
This is where threading comes in. Threading will allow you to process X number of lines at a time in parallel. This means that you can process a whole lot more data much quicker.
Fortunately for us, Python 3 provides some very useful libraries for handling Threads and making sure that everything starts and stops as intended.
Since threads are spun up and run asynchronously we will need to add a bit of extra code that will allow us to control the number of threads active at a given time, and also monitor all of our active threads and wait for them all to finish before allowing our program to close.
To do this we will use something called a "Thread Pool".
To help you to understand what a thread pool is a why it is useful, I will use the following analogy.
Image a busy bar at peek hours. There is a huge line out the door of people who are trying to get in to get a drink. There are only 5 or 6 bartenders on staff tonight and it was agreed before hand that there can only be a max of 50 people at the bar at a time otherwise the bartenders will become too overwhelmed to server everyone. The bouncer's job outside is to control the flow of people into the bar. He constantly monitors the number of people at the bar and only allows another person in to be served when the number of people at the bar is less than 50.
This helps to illustrate the problem that we face with multi threading. If we spin up an unlimited amount of threads, then we won't have the resources to serve all the function calls. This will result in errors and information not being processed correctly. Just to draw a direct comparison, the bar tenders are analogous to the function being run on each line. The customers are the number of threads and the bouncer is the pool manager. He is managing a pool of 50 people (threads) and never letting there be any more than 50 people (active threads) be served at any one time.
Now that you know what a thread pool is and why it is important, let's rewrite the above code using a thread pool.
Code
from concurrent.futures import ThreadPoolExecutor
def doStuff(line):
# Do stuff to line
return line
with open("big_test_file.txt") as btf:
thread_pool = []
executor = ThreadPoolExecutor(max_workers=10)
for line in btf:
thread = executor.submit(doStuff, line)
thread_pool.append(thread)
for active_thread in thread_pool:
while not active_thread.done():
pass
print(active_thread.result())As you can see, we didn't have to change much. All that really changed is how we are now calling doStuff()
We import a class called ThreadPoolExecutor that will handle the thread pool for us. Then instead of calling doStuff() directly we pass the function and the arguments to a method called "submit()" in the executor object. This adds a new thread to the pool that will be executed when < 5 threads are active.
The final for loop allows us to check for when a thread is completed. We iterate over the existing threads in the pool (thread_pool) and check if it is done using the .done() method. While the thread has not completed we wait. When the while condition is no longer satisfied we print the results and print the results of that thread.
And that's it! This is an incredibly useful too to have in your tool belt. It took me a while to fully understand but once I did I was able to use it everywhere. Some common issues people run into is synchronization of information and excess resource consumption. Managing threads and determine why they failed and when is also a common challenge. That is all for another post though.
I hope this helped! If you have any more questions feel free to ask. I am happy to help.
![[Image: logo.png]](https://proxyorbit.com/static/logo.png){kind=logo}
![[Image: TEGE4Zx.png]](https://i.imgur.com/TEGE4Zx.png)
