memoryError in FpxImageUploadPlugin.py

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

memoryError in FpxImageUploadPlugin.py

Gábor Farkas
hi,

we have a python webserver that handles files uploaded by the users.
some of those files cause a MemoryError in PIL.
(unfortunately i do not have such a file). as you can see, it happens
in the FpxImagPlugin:
(pil 1.1.7)

   im = Image.open(original)
 File "/usr/local/lib/python2.6/site-packages/PIL/Image.py", line 1976, in open
   return factory(fp, filename)
 File "/usr/local/lib/python2.6/site-packages/PIL/ImageFile.py", line
91, in __init__
   self._open()
 File "/usr/local/lib/python2.6/site-packages/PIL/FpxImagePlugin.py",
line 61, in _open
   self.ole = OleFileIO(self.fp)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
260, in __init__
   self.open(filename)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
289, in open
   self.loadfat(header)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
309, in loadfat
   s = self.getsect(ix)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
324, in getsect
   self.fp.seek(512 + self.sectorsize * sect)
MemoryError

any ideas what might cause this?

alternatively, is there a way to tell PIL to only try to handle a
given list of file-types?
for example, in my cases, i only care about JPG/PNG/GIF... so i do not need PIL
to try to find out if the file is in other formats.. is there any way
do a fileformat-whitelist?

thanks,
gabor
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: memoryError in FpxImageUploadPlugin.py

pontissimo
> i only care about JPG/PNG/GIF...
> is there any way do a fileformat-whitelist?

Seems almost too obvious to mention, but you could do something like:

fname = "something.jpg"

image_extensions = (".jpg", ".png", ".gif")

extension = os.path.splitext(fname)[1].lower()

if extension in image_extensions:

    print "its an image!"

    do_some_stuff(fname)





2011/4/15 Gábor Farkas <[hidden email]>
hi,

we have a python webserver that handles files uploaded by the users.
some of those files cause a MemoryError in PIL.
(unfortunately i do not have such a file). as you can see, it happens
in the FpxImagPlugin:
(pil 1.1.7)

  im = Image.open(original)
 File "/usr/local/lib/python2.6/site-packages/PIL/Image.py", line 1976, in open
  return factory(fp, filename)
 File "/usr/local/lib/python2.6/site-packages/PIL/ImageFile.py", line
91, in __init__
  self._open()
 File "/usr/local/lib/python2.6/site-packages/PIL/FpxImagePlugin.py",
line 61, in _open
  self.ole = OleFileIO(self.fp)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
260, in __init__
  self.open(filename)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
289, in open
  self.loadfat(header)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
309, in loadfat
  s = self.getsect(ix)
 File "/usr/local/lib/python2.6/site-packages/PIL/OleFileIO.py", line
324, in getsect
  self.fp.seek(512 + self.sectorsize * sect)
MemoryError

any ideas what might cause this?

alternatively, is there a way to tell PIL to only try to handle a
given list of file-types?
for example, in my cases, i only care about JPG/PNG/GIF... so i do not need PIL
to try to find out if the file is in other formats.. is there any way
do a fileformat-whitelist?

thanks,
gabor
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig


_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: memoryError in FpxImageUploadPlugin.py

Gábor Farkas
2011/4/16 Alec Bennett <[hidden email]>:

>> i only care about JPG/PNG/GIF...
>> is there any way do a fileformat-whitelist?
>
> Seems almost too obvious to mention, but you could do something like:
> fname = "something.jpg"
> image_extensions = (".jpg", ".png", ".gif")
> extension = os.path.splitext(fname)[1].lower()
> if extension in image_extensions:
>     print "its an image!"
>     do_some_stuff(fname)

is PIL's image-format-detection purely filename-extension based?

i mean, what happens if i take the problematic file, and rename it
to have a jpeg-extension... will PIL not try the other file-format-plugins,
when jpeg fails?

gabor
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: memoryError in FpxImageUploadPlugin.py

Charlie Clark-6
Am 16.04.2011, 17:51 Uhr, schrieb Gábor Farkas <[hidden email]>:

> is PIL's image-format-detection purely filename-extension based?
> i mean, what happens if i take the problematic file, and rename it
> to have a jpeg-extension... will PIL not try the other  
> file-format-plugins,
> when jpeg fails?

No, PIL won'.t It isn't brilliant but that's how it works. Try and think  
through the converse: this could be JPEG but I'll work my way through the  
alternatives till I'm sure. In your situation it seems perfectly  
reasonable to expect the MIME-type to be available, so you can enforce it  
in the form handling.

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: memoryError in FpxImageUploadPlugin.py

pontissimo
Agreed in a web form situation it seems reasonable to expect the file to have the correct file extension. And in fact that's the usual safeguard against people uploading php scripts or whatever containing malicious code.

But if you really want to be able to detect file type independently of the extension you might research reading the file header. (that's a guess, I'm not saying that will definitely work).




Sent from my payphone

On Apr 17, 2011, at 12:13 AM, "Charlie Clark" <[hidden email]> wrote:

> Am 16.04.2011, 17:51 Uhr, schrieb Gábor Farkas <[hidden email]>:
>
>> is PIL's image-format-detection purely filename-extension based?
>> i mean, what happens if i take the problematic file, and rename it
>> to have a jpeg-extension... will PIL not try the other file-format-plugins,
>> when jpeg fails?
>
> No, PIL won'.t It isn't brilliant but that's how it works. Try and think through the converse: this could be JPEG but I'll work my way through the alternatives till I'm sure. In your situation it seems perfectly reasonable to expect the MIME-type to be available, so you can enforce it in the form handling.
>
> Charlie
> --
> Charlie Clark
> Managing Director
> Clark Consulting & Research
> German Office
> Helmholtzstr. 20
> Düsseldorf
> D- 40215
> Tel: +49-211-600-3657
> Mobile: +49-178-782-6226
> _______________________________________________
> Image-SIG maillist  -  [hidden email]
> http://mail.python.org/mailman/listinfo/image-sig
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: memoryError in FpxImageUploadPlugin.py

Edward Cannon
One of the things you might want to consider is that users are
uploading either large or corrupted images. This might cause problems.
Also if something else on the server is not playing nice this could
cause problems. I would try wrapping the open call in a try... except
loop. That way you could handle the error gracefully, perhaps by
prompting the user to wait or try a smaller image.

AFIK PIL does not look at file extensions, but at magic numbers when
opening a file, it uses extensions primarily when saving files.
Edward Cannon
Unicorn School

On Sun, Apr 17, 2011 at 11:09 AM, Alec Bennett <[hidden email]> wrote:

> Agreed in a web form situation it seems reasonable to expect the file to have the correct file extension. And in fact that's the usual safeguard against people uploading php scripts or whatever containing malicious code.
>
> But if you really want to be able to detect file type independently of the extension you might research reading the file header. (that's a guess, I'm not saying that will definitely work).
>
>
>
>
> Sent from my payphone
>
> On Apr 17, 2011, at 12:13 AM, "Charlie Clark" <[hidden email]> wrote:
>
>> Am 16.04.2011, 17:51 Uhr, schrieb Gábor Farkas <[hidden email]>:
>>
>>> is PIL's image-format-detection purely filename-extension based?
>>> i mean, what happens if i take the problematic file, and rename it
>>> to have a jpeg-extension... will PIL not try the other file-format-plugins,
>>> when jpeg fails?
>>
>> No, PIL won'.t It isn't brilliant but that's how it works. Try and think through the converse: this could be JPEG but I'll work my way through the alternatives till I'm sure. In your situation it seems perfectly reasonable to expect the MIME-type to be available, so you can enforce it in the form handling.
>>
>> Charlie
>> --
>> Charlie Clark
>> Managing Director
>> Clark Consulting & Research
>> German Office
>> Helmholtzstr. 20
>> Düsseldorf
>> D- 40215
>> Tel: +49-211-600-3657
>> Mobile: +49-178-782-6226
>> _______________________________________________
>> Image-SIG maillist  -  [hidden email]
>> http://mail.python.org/mailman/listinfo/image-sig
> _______________________________________________
> Image-SIG maillist  -  [hidden email]
> http://mail.python.org/mailman/listinfo/image-sig
>
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig