can't use multiprocessing with class factory?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

can't use multiprocessing with class factory?

AlanIsaac
Can the below example be fixed to work?
Thanks,
Alan Isaac

import multiprocessing as mp

class Test(object):
    pass

def class_factory(x):
    class ConcreteTest(Test):
        _x = x
    return ConcreteTest

def f(cls):
    print cls._x

if __name__ == '__main__':
    pool = mp.Pool(2)
    pool.map(f, [class_factory(i) for i in range(4)])

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: can't use multiprocessing with class factory?

Robert Kern-2
On 1/28/11 1:02 PM, Alan wrote:

> Can the below example be fixed to work?
> Thanks,
> Alan Isaac
>
> import multiprocessing as mp
>
> class Test(object):
>      pass
>
> def class_factory(x):
>      class ConcreteTest(Test):
>          _x = x
>      return ConcreteTest
>
> def f(cls):
>      print cls._x
>
> if __name__ == '__main__':
>      pool = mp.Pool(2)
>      pool.map(f, [class_factory(i) for i in range(4)])

Send the (pickleable) factory and the arguments used to construct the instance,
not the unpickleable instance itself.

def g(factory, i):
     cls = factory(i)
     print cls._x

if __name__ == '__main__':
       pool = mp.Pool(2)
       pool.map(g, zip([class_factory] * 4, range(4)))

By the way, when asking for help like this, show us what your code did and
describe what results you want. It can often be hard to figure out exactly what
you mean by "work".

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: can't use multiprocessing with class factory?

Daniel Urban-2
In reply to this post by AlanIsaac
On Fri, Jan 28, 2011 at 20:02, Alan <[hidden email]> wrote:

> Can the below example be fixed to work?
> Thanks,
> Alan Isaac
>
> import multiprocessing as mp
>
> class Test(object):
>    pass
>
> def class_factory(x):
>    class ConcreteTest(Test):
>        _x = x
>    return ConcreteTest
>
> def f(cls):
>    print cls._x
>
> if __name__ == '__main__':
>    pool = mp.Pool(2)
>    pool.map(f, [class_factory(i) for i in range(4)])

Only classes defined on the top level of a module are picklable (see
http://docs.python.org/dev/py3k/library/pickle#what-can-be-pickled-and-unpickled
). The collections.namedtuple class factory function works around this
limitation by setting the __module__  attribute of the created class,
but I'm not sure if this solution can be used in this case.


Daniel
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: can't use multiprocessing with class factory?

hidura
In reply to this post by AlanIsaac
What is the output?

2011/1/28, Alan <[hidden email]>:

> Can the below example be fixed to work?
> Thanks,
> Alan Isaac
>
> import multiprocessing as mp
>
> class Test(object):
>     pass
>
> def class_factory(x):
>     class ConcreteTest(Test):
>         _x = x
>     return ConcreteTest
>
> def f(cls):
>     print cls._x
>
> if __name__ == '__main__':
>     pool = mp.Pool(2)
>     pool.map(f, [class_factory(i) for i in range(4)])
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

--
Enviado desde mi dispositivo móvil

Diego I. Hidalgo D.
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: can't use multiprocessing with class factory?

Robert Kern-2
In reply to this post by Daniel Urban-2
On 1/28/11 1:25 PM, Daniel Urban wrote:

> Only classes defined on the top level of a module are picklable (see
> http://docs.python.org/dev/py3k/library/pickle#what-can-be-pickled-and-unpickled
> ). The collections.namedtuple class factory function works around this
> limitation by setting the __module__  attribute of the created class,
> but I'm not sure if this solution can be used in this case.

namedtuple's trick only works when you assign the created class to a name at the
module level. E.g.

   MyFancyTuple = collections.namedtuple(...)

The trick won't work for "anonymous" classes like the above use case.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: can't use multiprocessing with class factory?

AlanIsaac
In reply to this post by AlanIsaac
On Jan 28, 2:23 pm, Robert Kern <[hidden email]> wrote:

> Send the (pickleable) factory and the arguments used to construct the instance,
> not the unpickleable instance itself.
>
> def g(factory, i):
>      cls = factory(i)
>      print cls._x
>
> if __name__ == '__main__':
>        pool = mp.Pool(2)
>        pool.map(g, zip([class_factory] * 4, range(4)))



If I change that to g((factory,i)) it does work.

Thanks!

Alan
--
http://mail.python.org/mailman/listinfo/python-list