Hey guys,

I'm working on a python script to parse through and search for keywords in my database collection. I'm curious what your thoughts are on using threading vs multiprocessing for handling the file reading?

multiprocessing for CPU-intensive processes,

If you are reading local files then MP ( uses concurrent.futures module )

(October 11, 2022, 09:28 PM)Gamma1225 Wrote: Hey guys,

I'm working on a python script to parse through and search for keywords in my database collection. I'm curious what your thoughts are on using threading vs multiprocessing for handling the file reading?

Threading can be good cause you share the same memory since its part of the same process, only split up into threads. So if youre wanting to pass data between threads, then this can be much more benificial. Its good for networking apps. So if you wanted to correlate certain keywords with one another then this could be the way to go. But i think in your case for just searching multi-processing is probably better. THis can run in parallel, so no need to worry about order of returned data.

If you are going to read all databases from the same disk, I think your fastest option is to use a single thread in a single process for that part, because sequential reading from a disk is much faster than random (yes, the files can be fragmented, but still...)

Based on the sqlite3 benchmarks multiprocessing is faster than multithreading for reading files. Its mostly because of the global interpreter lock in the python multithreading library

damn hard stuff