weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

kirby urner-4
Here we are in an Eclipse pydev console, running Python 3.1.2.  For
the most part, everything is working great.

However...

>>> import sys; print('%s %s' % (sys.executable or sys.platform, sys.version))
C:\Python31\python.exe 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC
v.1500 32 bit (Intel)]

>>> import pickle
>>> class Example:
...     def __init__(self):
...         self.name = "Hello"
...     def __repr__(self):
...         return "an Example object named {}".format(self.name)
...
...
>>> obj = Example()
>>> obj
an Example object named Hello

Note that I'm opening in binary, like I'm supposed to with this
latest protocol:

>>> f = open("testpickle.pkl",'wb')

Should be able to do this, no problemo:

>>> pickle.dump(obj, f)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Python31\lib\pickle.py", line 1354, in dump
    Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'Example'>: attribute
lookup builtins.Example failed

The above works fine in "naked Python" 3.1.2 by the way.
So this could be a problem with Eclipse / Pydev and/or
user error.  What am I missing?

Just normal data structures work:

>>> test = [1,2,3]
>>> pickle.dump(test,f)
>>> f.close()
>>>

Any other Eclipse users out there who can at least duplicate this
weirdness?

Kirby Urner
in Portland "Keep Portland Weird" Oregon

--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

Peter Otten
[hidden email] wrote:

> Here we are in an Eclipse pydev console, running Python 3.1.2.  For
> the most part, everything is working great.
>
> However...
>
>>>> import sys; print('%s %s' % (sys.executable or sys.platform,
>>>> sys.version))
> C:\Python31\python.exe 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC
> v.1500 32 bit (Intel)]
>
>>>> import pickle
>>>> class Example:
> ...     def __init__(self):
> ...         self.name = "Hello"
> ...     def __repr__(self):
> ...         return "an Example object named {}".format(self.name)
> ...
> ...
>>>> obj = Example()
>>>> obj
> an Example object named Hello
>
> Note that I'm opening in binary, like I'm supposed to with this
> latest protocol:
>
>>>> f = open("testpickle.pkl",'wb')
>
> Should be able to do this, no problemo:
>
>>>> pickle.dump(obj, f)
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "C:\Python31\lib\pickle.py", line 1354, in dump
>     Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute
> lookup builtins.Example failed
>
> The above works fine in "naked Python" 3.1.2 by the way.
> So this could be a problem with Eclipse / Pydev and/or
> user error.  What am I missing?
>
> Just normal data structures work:
>
>>>> test = [1,2,3]
>>>> pickle.dump(test,f)
>>>> f.close()
>>>>
>
> Any other Eclipse users out there who can at least duplicate this
> weirdness?

I can provoke the error in "naked" Python 3 by changing the
Example.__module__ attribute:

Python 3.1.1+ (r311:74480, Nov  2 2009, 15:45:00)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> class Example:
...     pass
...
>>> pickle.dumps(Example())
b'\x80\x03c__main__\nExample\nq\x00)\x81q\x01}q\x02b.'
>>> Example.__module__ = "builtins"
>>> pickle.dumps(Example())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.1/pickle.py", line 1358, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'Example'>: attribute lookup
builtins.Example failed

What's the value of __module__ when you run your code in Eclipse?

Peter
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

kirby urner-4
On Jun 4, 9:47 am, Peter Otten <[hidden email]> wrote:

> I can provoke the error in "naked" Python 3 by changing the
> Example.__module__ attribute:
>
> Python 3.1.1+ (r311:74480, Nov  2 2009, 15:45:00)
> [GCC 4.4.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.

> >>> import pickle
> >>> class Example:
>
> ...     pass
> ...
> >>> pickle.dumps(Example())
>
> b'\x80\x03c__main__\nExample\nq\x00)\x81q\x01}q\x02b.'

> >>> Example.__module__ = "builtins"
> >>> pickle.dumps(Example())
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python3.1/pickle.py", line 1358, in dumps
>     Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute lookup
> builtins.Example failed
>
> What's the value of __module__ when you run your code in Eclipse?
>
> Peter

Thank you for replying.

Here's from Eclipse console:

>>> Example.__module__
'builtins'

>>> __name__
'builtins'

Duplicating your result in naked Python:

Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> class Example:
        pass

>>> import pickle
>>> Example.__module__
'__main__'
>>> f = open('testpickle.pkl','wb')
>>> obj = Example()
>>> obj
<__main__.Example object at 0x02A26690>
>>> pickle.dump(obj, f)

>>> Example.__module__ = 'builtins'
>>> obj2 = Example()

>>> pickle.dump(obj2, f)
Traceback (most recent call last):
  File "<pyshell#12>", line 1, in <module>
    pickle.dump(obj2, f)
  File "C:\Python31\lib\pickle.py", line 1354, in dump
    Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'Example'>: attribute
lookup builtins.Example failed

So what if I'm in an Eclipse pydev console and
change the Example.__module__ to '__main__'....

>>> import sys; print('%s %s' % (sys.executable or sys.platform, sys.version))
C:\Python31\python.exe 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC
v.1500 32 bit (Intel)]
>>> import pickle
>>> class Example:
...     pass
...
>>> Example.__module__
'builtins'
>>> Example.__module__ = '__main__'
>>> obj = Example()
>>> obj
<__main__.Example object at 0x029E8FD0>
>>> f = open('testpickle.pkl','wb')
>>> pickle.dump(obj, f)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Python31\lib\pickle.py", line 1354, in dump
    Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class '__main__.Example'>:
attribute lookup __main__.Example failed
>>>

Dang.

Any insights?

Kirby
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

Peter Otten
[hidden email] wrote:

> On Jun 4, 9:47 am, Peter Otten <[hidden email]> wrote:
>
>> I can provoke the error in "naked" Python 3 by changing the
>> Example.__module__ attribute:
>>
>> Python 3.1.1+ (r311:74480, Nov  2 2009, 15:45:00)
>> [GCC 4.4.1] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>
>> >>> import pickle
>> >>> class Example:
>>
>> ...     pass
>> ...
>> >>> pickle.dumps(Example())
>>
>> b'\x80\x03c__main__\nExample\nq\x00)\x81q\x01}q\x02b.'
>
>> >>> Example.__module__ = "builtins"
>> >>> pickle.dumps(Example())
>>
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/usr/lib/python3.1/pickle.py", line 1358, in dumps
>> Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
>> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute lookup
>> builtins.Example failed
>>
>> What's the value of __module__ when you run your code in Eclipse?
>>
>> Peter
>
> Thank you for replying.
>
> Here's from Eclipse console:
>
>>>> Example.__module__
> 'builtins'
>
>>>> __name__
> 'builtins'
>
> Duplicating your result in naked Python:
>
> Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "copyright", "credits" or "license()" for more information.
>>>> class Example:
> pass
>
>>>> import pickle
>>>> Example.__module__
> '__main__'
>>>> f = open('testpickle.pkl','wb')
>>>> obj = Example()
>>>> obj
> <__main__.Example object at 0x02A26690>
>>>> pickle.dump(obj, f)
>
>>>> Example.__module__ = 'builtins'
>>>> obj2 = Example()
>
>>>> pickle.dump(obj2, f)
> Traceback (most recent call last):
>   File "<pyshell#12>", line 1, in <module>
>     pickle.dump(obj2, f)
>   File "C:\Python31\lib\pickle.py", line 1354, in dump
>     Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute
> lookup builtins.Example failed
>
> So what if I'm in an Eclipse pydev console and
> change the Example.__module__ to '__main__'....
>
>>>> import sys; print('%s %s' % (sys.executable or sys.platform,
>>>> sys.version))
> C:\Python31\python.exe 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC
> v.1500 32 bit (Intel)]
>>>> import pickle
>>>> class Example:
> ...     pass
> ...
>>>> Example.__module__
> 'builtins'
>>>> Example.__module__ = '__main__'
>>>> obj = Example()
>>>> obj
> <__main__.Example object at 0x029E8FD0>
>>>> f = open('testpickle.pkl','wb')
>>>> pickle.dump(obj, f)
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "C:\Python31\lib\pickle.py", line 1354, in dump
>     Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
> _pickle.PicklingError: Can't pickle <class '__main__.Example'>:
> attribute lookup __main__.Example failed
>>>>
>
> Dang.
>
> Any insights?

Sorry, no. Consider a bug report to the pydev project.

Peter
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

Fabio Zadrozny-2
On Mon, Jun 7, 2010 at 6:30 AM, Peter Otten <[hidden email]> wrote:

> [hidden email] wrote:
>
>> On Jun 4, 9:47 am, Peter Otten <[hidden email]> wrote:
>>
>>> I can provoke the error in "naked" Python 3 by changing the
>>> Example.__module__ attribute:
>>>
>>> Python 3.1.1+ (r311:74480, Nov  2 2009, 15:45:00)
>>> [GCC 4.4.1] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>
>>> >>> import pickle
>>> >>> class Example:
>>>
>>> ...     pass
>>> ...
>>> >>> pickle.dumps(Example())
>>>
>>> b'\x80\x03c__main__\nExample\nq\x00)\x81q\x01}q\x02b.'
>>
>>> >>> Example.__module__ = "builtins"
>>> >>> pickle.dumps(Example())
>>>
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> File "/usr/lib/python3.1/pickle.py", line 1358, in dumps
>>> Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
>>> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute lookup
>>> builtins.Example failed
>>>
>>> What's the value of __module__ when you run your code in Eclipse?
>>>
>>> Peter
>>
>> Thank you for replying.
>>
>> Here's from Eclipse console:
>>
>>>>> Example.__module__
>> 'builtins'
>>
>>>>> __name__
>> 'builtins'
>>
>> Duplicating your result in naked Python:
>>
>> Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
>> (Intel)] on win32
>> Type "copyright", "credits" or "license()" for more information.
>>>>> class Example:
>> pass
>>
>>>>> import pickle
>>>>> Example.__module__
>> '__main__'
>>>>> f = open('testpickle.pkl','wb')
>>>>> obj = Example()
>>>>> obj
>> <__main__.Example object at 0x02A26690>
>>>>> pickle.dump(obj, f)
>>
>>>>> Example.__module__ = 'builtins'
>>>>> obj2 = Example()
>>
>>>>> pickle.dump(obj2, f)
>> Traceback (most recent call last):
>>   File "<pyshell#12>", line 1, in <module>
>>     pickle.dump(obj2, f)
>>   File "C:\Python31\lib\pickle.py", line 1354, in dump
>>     Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
>> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute
>> lookup builtins.Example failed
>>
>> So what if I'm in an Eclipse pydev console and
>> change the Example.__module__ to '__main__'....
>>
>>>>> import sys; print('%s %s' % (sys.executable or sys.platform,
>>>>> sys.version))
>> C:\Python31\python.exe 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC
>> v.1500 32 bit (Intel)]
>>>>> import pickle
>>>>> class Example:
>> ...     pass
>> ...
>>>>> Example.__module__
>> 'builtins'
>>>>> Example.__module__ = '__main__'
>>>>> obj = Example()
>>>>> obj
>> <__main__.Example object at 0x029E8FD0>
>>>>> f = open('testpickle.pkl','wb')
>>>>> pickle.dump(obj, f)
>> Traceback (most recent call last):
>>   File "<console>", line 1, in <module>
>>   File "C:\Python31\lib\pickle.py", line 1354, in dump
>>     Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
>> _pickle.PicklingError: Can't pickle <class '__main__.Example'>:
>> attribute lookup __main__.Example failed
>>>>>
>>
>> Dang.
>>
>> Any insights?
>
> Sorry, no. Consider a bug report to the pydev project.
>


I've taken a look at the pydev code and the issue seems to be in
python itself. See: http://bugs.python.org/issue8943

Cheers,

Fabio
--
http://mail.python.org/mailman/listinfo/python-list
Reply | Threaded
Open this post in threaded view
|

Re: weird pickle behavior in Python 3.1.2 + Eclipse 3.5.2

Peter Otten
Fabio Zadrozny wrote:

> On Mon, Jun 7, 2010 at 6:30 AM, Peter Otten <[hidden email]> wrote:
>> [hidden email] wrote:
>>
>>> On Jun 4, 9:47 am, Peter Otten <[hidden email]> wrote:
>>>
>>>> I can provoke the error in "naked" Python 3 by changing the
>>>> Example.__module__ attribute:
>>>>
>>>> Python 3.1.1+ (r311:74480, Nov  2 2009, 15:45:00)
>>>> [GCC 4.4.1] on linux2
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>> >>> import pickle
>>>> >>> class Example:
>>>>
>>>> ...     pass
>>>> ...
>>>> >>> pickle.dumps(Example())
>>>>
>>>> b'\x80\x03c__main__\nExample\nq\x00)\x81q\x01}q\x02b.'
>>>
>>>> >>> Example.__module__ = "builtins"
>>>> >>> pickle.dumps(Example())
>>>>
>>>> Traceback (most recent call last):
>>>> File "<stdin>", line 1, in <module>
>>>> File "/usr/lib/python3.1/pickle.py", line 1358, in dumps
>>>> Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
>>>> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute lookup
>>>> builtins.Example failed
>>>>
>>>> What's the value of __module__ when you run your code in Eclipse?
>>>>
>>>> Peter
>>>
>>> Thank you for replying.
>>>
>>> Here's from Eclipse console:
>>>
>>>>>> Example.__module__
>>> 'builtins'
>>>
>>>>>> __name__
>>> 'builtins'
>>>
>>> Duplicating your result in naked Python:
>>>
>>> Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit
>>> (Intel)] on win32
>>> Type "copyright", "credits" or "license()" for more information.
>>>>>> class Example:
>>> pass
>>>
>>>>>> import pickle
>>>>>> Example.__module__
>>> '__main__'
>>>>>> f = open('testpickle.pkl','wb')
>>>>>> obj = Example()
>>>>>> obj
>>> <__main__.Example object at 0x02A26690>
>>>>>> pickle.dump(obj, f)
>>>
>>>>>> Example.__module__ = 'builtins'
>>>>>> obj2 = Example()
>>>
>>>>>> pickle.dump(obj2, f)
>>> Traceback (most recent call last):
>>> File "<pyshell#12>", line 1, in <module>
>>> pickle.dump(obj2, f)
>>> File "C:\Python31\lib\pickle.py", line 1354, in dump
>>> Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
>>> _pickle.PicklingError: Can't pickle <class 'Example'>: attribute
>>> lookup builtins.Example failed
>>>
>>> So what if I'm in an Eclipse pydev console and
>>> change the Example.__module__ to '__main__'....
>>>
>>>>>> import sys; print('%s %s' % (sys.executable or sys.platform,
>>>>>> sys.version))
>>> C:\Python31\python.exe 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC
>>> v.1500 32 bit (Intel)]
>>>>>> import pickle
>>>>>> class Example:
>>> ...     pass
>>> ...
>>>>>> Example.__module__
>>> 'builtins'
>>>>>> Example.__module__ = '__main__'
>>>>>> obj = Example()
>>>>>> obj
>>> <__main__.Example object at 0x029E8FD0>
>>>>>> f = open('testpickle.pkl','wb')
>>>>>> pickle.dump(obj, f)
>>> Traceback (most recent call last):
>>> File "<console>", line 1, in <module>
>>> File "C:\Python31\lib\pickle.py", line 1354, in dump
>>> Pickler(file, protocol, fix_imports=fix_imports).dump(obj)
>>> _pickle.PicklingError: Can't pickle <class '__main__.Example'>:
>>> attribute lookup __main__.Example failed
>>>>>>
>>>
>>> Dang.
>>>
>>> Any insights?
>>
>> Sorry, no. Consider a bug report to the pydev project.
>
> I've taken a look at the pydev code and the issue seems to be in
> python itself. See: http://bugs.python.org/issue8943

Thanks for taking a look!

While this is being worked on you could put a workaround into pydev: you'd
just have to pass the global dict of a real module to InteractiveConsole:

$ cat fast_test.py
import sys
from code import InteractiveConsole

if "-m" in sys.argv:
    console = InteractiveConsole(globals())
else:
    console = InteractiveConsole()

contents = '''
import pickle
class Example:
    def __init__(self):
        self.name = "Hello"
    def __repr__(self):
        return "an Example object named " + self.name


obj = Example()
obj

f = open("testpickle.pkl", 'wb')
pickle.dump(obj, f)
f.close()

pickle.load(open("testpickle.pkl", "rb"))
'''

for line in contents.splitlines():
    console.push(line)
$ python2.4 fast_test.py -m
an Example object named Hello
an Example object named Hello
$ python2.5 fast_test.py -m
an Example object named Hello
an Example object named Hello
$ python2.6 fast_test.py -m
an Example object named Hello
an Example object named Hello
$ python3.1 fast_test.py -m
an Example object named Hello
an Example object named Hello
$

Peter

--
http://mail.python.org/mailman/listinfo/python-list