Python Multiprocessing Takes More Time

I have server with 12 cores and 28GB RAM. I am running two versions of Python; one with multiprocessing and another sequential. I expect the Multiprocessing.py to finish early com

Solution 1:

The problem is that too little work is being done relative to the IPC communication overhead.

The cube function isn't a good candidate for multiprocessing speedup. Try something "more interesting" like function that computes the sum of cube for 1 to n or somesuch:

import os, multiprocessing, time

defsum_of_cubes(n):
    returnsum(x**3for x inrange(n))

if __name__ == '__main__':

    from multiprocessing.pool import ThreadPool as Pool

    pool = Pool(25)

    start = time.time()
    print(pool.map(sum_of_cubes, range(1000, 100000, 1000)))
    end = time.time()
    print(end - start)

The general rules are:

• don't start more pools than your cores can benefit from
• don't pass in a lot of data or return a lot of data (too much IPC load)
• do significant work in the process relative to the IPC overhead.

Solution 2:

You shouldn't be starting a process for each multiplication. Start 12 processes and pass each one the numbers or hand out the numbers at process creation.

If you profile that I'm fairly certain you'll find all your time spent in process creation and clean up.

ALSO: I've done testing on how many processes to run vs core count and the optimum depends on architecture (e.g. some intel chips have 2x threads per core) and operating system (Linux seems to handle it better than Windows). If you're on windows I'd advise trying process counts of 0.8-2.2x core count. On Linux you can do more.

Solution 3:

Would you like to try pool? For example, the following should work:

from multiprocessing import Pool

p = Pool(12)
Results = p.map(cube, range(5000))