Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Here's a use case: I was training a neutral net, and wanted to do some preprocessing (similar to image resizing, but without an existing C function). Inputs are batched, so the preprocessing is trivially parallelizable. I tried to multithread it in python, and got no speedup at all.

That was a really sad moment, and I've never felt good about python since.



Yeah, I had something similar.

I wanted "as you wait for the GPU to churn through this batch, start reading the next batch from disk & preprocessing it on the CPU"

Getting this to work turned out so ass backwards it made me sad

Also I pity the fool who tries to connect a debugger to code using multiprocessing.Pool()...


> Also I pity the fool who tries to connect a debugger to code using multiprocessing.Pool()

PyCharm works fine


Yeah, had the exact problem.

I solved it by using python threads and C function (cv2.resize)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: