IPython parallel and objects

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

IPython parallel and objects

thwiouz
Hi guys,

I'd like to parallelize the following loop:

m1 = np.arange(0, 10, 100)
m2 = np.arange(0, 10, 100)

for m_ in m1:
    for n_2 in m2:
        tmp = optimizer.compute_expectation([m_, n_])
        res.append(tmp)

with type(optimizer) some class that I've defined. How could I handle it?

I tried to do the same as in http://stackoverflow.com/questions/9363118/parallelise-nested-for-loop-in-ipython so I created a lambda function f

f = lambda x, y: optimizer.compute_expectation([x, y])

but it's not working with a load balanced view, saying that optimizer is not defined.

Could someone help?

Thanks,
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel and objects

Moritz Beber
Hi,

On Wed, Jan 28, 2015 at 1:40 PM, thwiouz <[hidden email]> wrote:
Hi guys,

I'd like to parallelize the following loop:

m1 = np.arange(0, 10, 100)
m2 = np.arange(0, 10, 100)

for m_ in m1:
    for n_2 in m2:
        tmp = optimizer.compute_expectation([m_, n_])
        res.append(tmp)

with type(optimizer) some class that I've defined. How could I handle it?

I tried to do the same as in
http://stackoverflow.com/questions/9363118/parallelise-nested-for-loop-in-ipython
so I created a lambda function f

f = lambda x, y: optimizer.compute_expectation([x, y])

but it's not working with a load balanced view, saying that optimizer is not
defined.

IPython only automatically transmits the function definition used in a `map` call or similar. Your optimizer is inside your lambda function and thus unknown on the remote kernels. In order to overcome this, take a look here:

http://ipython.org/ipython-doc/2/parallel/parallel_multiengine.html#remote-function-decorators

HTH,
Moritz


_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: IPython parallel and objects

Michael McKerns-2
If you are looking to convert nested for-loops to hierarchical parallel maps,
this is the kind of thing that `pathos` is built to do.  If you are
interested
in an optimizer that can utilize hierarchical parallelism to compute
expectation
values, this is the kind of thing that `mystic` is designed to do.

If you want to get the code, the best place to get them is here:
https://github.com/uqfoundation

Examples:
http://stackoverflow.com/questions/28203774/how-to-do-hierarchical-parallelism-in-ipython-parallel
http://stackoverflow.com/questions/27751023/converting-serial-to-parallel-to-multi-machine

Neither `pathos` nor `mystic` use IPython-parallel, but I hope to make them
more compatible with it in the near future.


> Hi,
>
> On Wed, Jan 28, 2015 at 1:40 PM, thwiouz <[hidden email]> wrote:
>
>> Hi guys,
>>
>> I'd like to parallelize the following loop:
>>
>> m1 = np.arange(0, 10, 100)
>> m2 = np.arange(0, 10, 100)
>>
>> for m_ in m1:
>>     for n_2 in m2:
>>         tmp = optimizer.compute_expectation([m_, n_])
>>         res.append(tmp)
>>
>> with type(optimizer) some class that I've defined. How could I handle
>> it?
>>
>> I tried to do the same as in
>>
>> http://stackoverflow.com/questions/9363118/parallelise-nested-for-loop-in-ipython
>> so I created a lambda function f
>>
>> f = lambda x, y: optimizer.compute_expectation([x, y])
>>
>> but it's not working with a load balanced view, saying that optimizer is
>> not
>> defined.
>>
>
> IPython only automatically transmits the function definition used in a
> `map` call or similar. Your optimizer is inside your lambda function and
> thus unknown on the remote kernels. In order to overcome this, take a look
> here:
>
> http://ipython.org/ipython-doc/2/parallel/parallel_multiengine.html#remote-function-decorators
>
> and here:
>
> http://ipython.org/ipython-doc/2/parallel/parallel_task.html#dependencies
>
> this could be helpful, too:
>
> http://ipython.org/ipython-doc/2/parallel/parallel_multiengine.html#moving-python-objects-around
>
>
> HTH,
> Moritz
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>


---

Mike McKerns
California Institute of Technology
TEL: (626)395-5773 or (626)590-8470
http://www.its.caltech.edu/~mmckerns
[hidden email]

_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev