Among the many libraries for parallel processing in python I chose what seemed to have one of the simpler interfaces - Parallel Python.
There is one trick that is not apparent from the docs: The function you call has to be 'self sufficient'. You basically have to write up the function you call as if it were a script. So the function has to have the imports etc within itself.
This is not a biggie, but it gets confusing until you figure it out.
Original code:
Parallelized code
There is one trick that is not apparent from the docs: The function you call has to be 'self sufficient'. You basically have to write up the function you call as if it were a script. So the function has to have the imports etc within itself.
This is not a biggie, but it gets confusing until you figure it out.
Original code:
# module x -----------------
import a
import b
def f1(g):
f2
f3
def f2():
def f3():
# Calling script
import x
x.f1(2)
x.f1(4)
x.f1(6)
x.f1(8)
Parallelized code
# module px ------------------------------
def pf(g)
import x <------ NOTE THIS. ALL IMPORTS AND OTHER FUNS HAVE TO BE WITHIN THIS FUN
x.f(g)
# New calling script
import pp
import px
ppservers = ()
job_server = pp.Server(ppservers=ppservers)
jobs = []
jobs.append(job_server.submit(px.pf, (2,))
jobs.append(job_server.submit(px.pf, (4,))
jobs.append(job_server.submit(px.pf, (6,))
jobs.append(job_server.submit(px.pf, (8,))
Comments
Post a Comment