Python Multiprocessing Blocks Indefinately In Waiter.acquire()
Solution 1:
Sorry for giving out the wrong answer. It's so irresponsible for not verify it. Here is the answer from me.
with Pool(processes=numParallelProcesses) as pool:
This line is wrong as with will call exit function not close. Here is exit function body:
def __exit__(self, exc_type, exc_val, exc_tb):
self.terminate()
All of the process will be terminated and never excuted. Code:
ps = [ res.get() for res in proclist ]
there is no timeout parameter. Here is the get function body:
def get(self, timeout=None):
self.wait(timeout)
if not self.ready():
raise TimeoutError
if self._success:
return self._value
else:
raise self._value
It will always wait if no timeout. That's why it hang.
You need to change
with Pool(processes=numParallelProcesses) as pool:
proclist = [ pool.apply_async(fitWithErr, args) for args in trialParameterList ]
to:
pool=Pool(processes=numParallelProcesses)
proclist = [ pool.apply_async(fitWithErr, args) for args in trialParameterList ]
pool.close()
Solution 2:
Indent
After all that, it was just that I didn't realize some code was not in the with
clause that was supposed to be. (Besides some typos and other bugs, which I've now fixed.) Intermezzo strikes again!
Thanks to Snowy for making me go through it a different way until I found my error. I it was just not clear what I intended to do. Snowy's ode is a perfectly valid and equivalent code. However, for the record, timeout
is not necessary. And, more importantly, with
is perfectly valid for Process if you use it correctly, as shown in the very first paragraph of the Python3.6.6 multiprocessing
documentation, which is where I got it. I just messed it up, somehow. The code I was trying to write was simply:
with Pool(processes=numParallelProcesses) as pool:
proclist = [ pool.apply_async(fitWithErr, args) for args in trialParameterList ]
ps = [ res.get() for res in proclist ]
ps = np.array(ps)
mean_pfit = np.mean(ps,0)
Works like I expected.
Post a Comment for "Python Multiprocessing Blocks Indefinately In Waiter.acquire()"