Quantcast

[IPython-User] ipyparallel: Can't pickly memoryview objects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[IPython-User] ipyparallel: Can't pickly memoryview objects

Florian Lindner
Hello,

I have this small sample program:

import ipyparallel as ipp, numpy as np

class X:
    def __init__(self, x, xs):
        self.A = np.full((10, 10), x)
        np.fill_diagonal(self.A, x*10)

    def norm(self):
        return np.linalg.norm(self.A)


def main():
    rc = ipp.Client()
    lview = rc.load_balanced_view()
    dview = rc[:]
    dview.push({"X" : X})
    dview.execute("import numpy as np")

    xs = np.linspace(1, 100, 10)
    sqxs = []

    sqxs = lview.map(lambda x: X(x, xs), xs)
    sqxs.wait()

    for i in sqxs:
        print(i.norm())

if __name__ == "__main__":
    main()


trying it to run gives TypeError: can't pickle memoryview objects. Probably because I provide the xs array to X.

How can I work around this situation?

Thanks,
Florian

Complete traceback:

Traceback (most recent call last):
  File "ipython_parallel.py", line 30, in <module>
    main()
  File "ipython_parallel.py", line 23, in main
    sqxs = lview.map(lambda x: X(x, xs), xs)
  File "<decorator-gen-136>", line 2, in map
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 50, in sync_results
    ret = f(self, *args, **kwargs)
  File "<decorator-gen-135>", line 2, in map
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 35, in save_ids
    ret = f(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 1109, in map
    return pf.map(*sequences)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py", line 285, in map
    return self(*sequences, __ipp_mapping=True)
  File "<decorator-gen-118>", line 2, in __call__
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py", line 76, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py", line 259, in __call__
    ar = view.apply(f, *args)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 211, in apply
    return self._really_apply(f, args, kwargs)
  File "<decorator-gen-134>", line 2, in _really_apply
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 47, in sync_results
    return f(self, *args, **kwargs)
  File "<decorator-gen-133>", line 2, in _really_apply
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 35, in save_ids
    ret = f(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 1037, in _really_apply
    metadata=metadata)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/client.py", line 1395, in send_apply_request
    item_threshold=self.session.item_threshold,
  File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py", line 166, in pack_apply_message
    serialize_object(arg, buffer_threshold, item_threshold) for arg in args))
  File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py", line 166, in <genexpr>
    serialize_object(arg, buffer_threshold, item_threshold) for arg in args))
  File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py", line 112, in serialize_object
    buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))
TypeError: can't pickle memoryview objects

_______________________________________________
IPython-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/ipython-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ipyparallel: Can't pickly memoryview objects

Fernando Perez
Hi Florian,

sorry, this list is mostly deprecated, with all IPython discussions now happening on a single list:


The issue is the closure over xs in your lambda; pickle doesn't know how to serialize closures.  You can make it work if you enable the cloudpickle serializer instead:

    dview.use_cloudpickle()


An interesting tidbit is that if you move the code in `main` out to the top-level of the script and manually push `xs` as well, the code works fine even without the cloudpickle change.  That's because names that require a lookup into globals() don't require the creation of a closure object, as explained in the link above.

Cheers,

f


On Thu, Mar 16, 2017 at 2:07 AM, Florian Lindner <[hidden email]> wrote:
Hello,

I have this small sample program:

import ipyparallel as ipp, numpy as np

class X:
    def __init__(self, x, xs):
        self.A = np.full((10, 10), x)
        np.fill_diagonal(self.A, x*10)

    def norm(self):
        return np.linalg.norm(self.A)


def main():
    rc = ipp.Client()
    lview = rc.load_balanced_view()
    dview = rc[:]
    dview.push({"X" : X})
    dview.execute("import numpy as np")

    xs = np.linspace(1, 100, 10)
    sqxs = []

    sqxs = lview.map(lambda x: X(x, xs), xs)
    sqxs.wait()

    for i in sqxs:
        print(i.norm())

if __name__ == "__main__":
    main()


trying it to run gives TypeError: can't pickle memoryview objects. Probably because I provide the xs array to X.

How can I work around this situation?

Thanks,
Florian

Complete traceback:

Traceback (most recent call last):
  File "ipython_parallel.py", line 30, in <module>
    main()
  File "ipython_parallel.py", line 23, in main
    sqxs = lview.map(lambda x: X(x, xs), xs)
  File "<decorator-gen-136>", line 2, in map
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 50, in sync_results
    ret = f(self, *args, **kwargs)
  File "<decorator-gen-135>", line 2, in map
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 35, in save_ids
    ret = f(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 1109, in map
    return pf.map(*sequences)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py", line 285, in map
    return self(*sequences, __ipp_mapping=True)
  File "<decorator-gen-118>", line 2, in __call__
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py", line 76, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/remotefunction.py", line 259, in __call__
    ar = view.apply(f, *args)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 211, in apply
    return self._really_apply(f, args, kwargs)
  File "<decorator-gen-134>", line 2, in _really_apply
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 47, in sync_results
    return f(self, *args, **kwargs)
  File "<decorator-gen-133>", line 2, in _really_apply
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 35, in save_ids
    ret = f(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/view.py", line 1037, in _really_apply
    metadata=metadata)
  File "/usr/lib/python3.6/site-packages/ipyparallel/client/client.py", line 1395, in send_apply_request
    item_threshold=self.session.item_threshold,
  File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py", line 166, in pack_apply_message
    serialize_object(arg, buffer_threshold, item_threshold) for arg in args))
  File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py", line 166, in <genexpr>
    serialize_object(arg, buffer_threshold, item_threshold) for arg in args))
  File "/usr/lib/python3.6/site-packages/ipyparallel/serialize/serialize.py", line 112, in serialize_object
    buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))
TypeError: can't pickle memoryview objects

_______________________________________________
IPython-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/ipython-user



--
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

_______________________________________________
IPython-User mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/ipython-user
Loading...