HTTPS and file uploads (a bug? Or is it just me?)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

HTTPS and file uploads (a bug? Or is it just me?)

Alan Pound
I now have a prototype site using a mixture of HTTP and HTTPS (for the
secure bits).  It all seems to be coming on fine, except with respect
to one current problem in file uploads.  I've reduced the code to the
example below (hopefully not having thrown bits of baby out with the
bathwater...)


import os, cherrypy

class Root:
    @cherrypy.expose
    def index(self):
        return """<html><body>
            <form action="upload" method="post" enctype="multipart/
form-data">
            filename: <input type="file" name="myFile"/><br/><input
type="submit"/>
            </form></body></html>"""

    @cherrypy.expose
    def upload(self, myFile):
        return """<html><body>myFile filename: %s<br/>myFile mime-
type: %s</body></html>
        """ % ( myFile.filename, myFile.type)

global_conf = {'global': {
                'server.socket_host': '0.0.0.0',
                'server.socket_port': 8080,
                #'server.ssl_certificate':  'prg/cms.aculab.com.crt',
                #'server.ssl_private_key':  'prg/cms.aculab.com.key',
                'log.screen': True}
              }

cherrypy.config.update( global_conf)
cherrypy.quickstart( Root())

Run exactly as shown with HTTP on 8080, I can upload files (well, they
get into CP and print the little summary, but my example makes no
effort to save 'myFile' from there...) - files of any practical
size...

If I comment out the two ssl_ config elements, then everything will
work just the same using HTTPS on 8080, but will only successfully
upload files < 8192 bytes.  A 100k file, for example causes the
following complaint (at the screen - the page itself never appears)...

<stacks of stack trace...>
  File "/usr/local/lib/python2.6/dist-packages/cherrypy/
_cpreqbody.py", line 560, in read
    data = self.fp.read(chunksize)
  File "/usr/local/lib/python2.6/dist-packages/cherrypy/wsgiserver/
__init__.py", line 265, in read
    data = self.rfile.read(size)
  File "/usr/local/lib/python2.6/dist-packages/cherrypy/wsgiserver/
__init__.py", line 932, in read
    assert n <= left, "recv(%d) returned %d bytes" % (left, n)
AssertionError: recv(8192) returned 16384 bytes

A 10.2k file causes a similar complaint that  10606 bytes have been
received....

Yet everything else I have done under SSL (downloads, for example)
seem to work just fine...

CP version 3.2.0rc1 on python 2.6.5 on Ubuntu 10.4.

Again, thanks for your help
Alan

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS and file uploads (a bug? Or is it just me?)

Alan Pound
A little more info...

I've spotted there is an open ticket on this very issue (#954 - about
a year old), so I'm not too hopeful of a quick resolution
here ;~(   However, a couple of additional observations:

i) I had OpenSSL v0.10 installed (ubuntu 10.4 considers this the
'current' version).  I've installed the latest 0.12a2, and *no
difference* that I can see...

ii) We have a Fedora 8 system here that doesn't exhibit the problem
(under apparently identical conditions!!!)

... perplexed ...

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Re: HTTPS and file uploads (a bug? Or is it just me?)

Iztok Kavkler-2
On 26.10.2010 18:18, alanp wrote:

> A little more info...
>
> I've spotted there is an open ticket on this very issue (#954 - about
> a year old), so I'm not too hopeful of a quick resolution
> here ;~(   However, a couple of additional observations:
>
> i) I had OpenSSL v0.10 installed (ubuntu 10.4 considers this the
> 'current' version).  I've installed the latest 0.12a2, and *no
> difference* that I can see...
>
> ii) We have a Fedora 8 system here that doesn't exhibit the problem
> (under apparently identical conditions!!!)
>
> ... perplexed ...
>

I don't know if this is the same problem, but: Cherrypy has serious
issues with timeouts when reading the body of a large post request. Just
try adding

server.socket_timeout = <some large number>

to the global section of your config.

There is more about it on the page

http://nickmurdoch.livejournal.com/382648.html

Best,
Iztok

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Re: HTTPS and file uploads (a bug? Or is it just me?)

Alan Pound
On Wed, 2010-10-27 at 13:25 +0200, Iztok Kavkler wrote:

> I don't know if this is the same problem, but: Cherrypy has serious
> issues with timeouts when reading the body of a large post request. Just
> try adding
>
> server.socket_timeout = <some large number>
>
> to the global section of your config.
>
> There is more about it on the page
>
> http://nickmurdoch.livejournal.com/382648.html

Hey thanks - that's a great tip.  

However, I'm certain it's not my issue....  I now have (what for the
moment appears to be) a decent work-around...  I'll post it back here
when I have more certainty (I've already eaten too many words in my life
to be in a hurry...)

Rgrds, Alan


--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS and file uploads (a bug? Or is it just me?)

Alan Pound
So I have a "work-around" (aka a really horrible cruft!)

The first observation is that someone put the assert in the read()
function "__init__.py" to detect precisely this problematic condition
- that the recv( 8192) call returns more than 8192... And yes, under
certain conditions, under ssl, it clearly does (but maybe not for
everybody!)

I put stack trace into the recv function and tested uploading a 10k
file, and it is clear that when read() calls self.recv(8192), the call
initially visits a recv() fn in the ssl library, which then calls down
to the __init__ recv() function *twice*, first getting 8192 bytes,
then getting a further 2k, all of which (10k) it then returns to the
read function.

So it seems to be the ssl library that isn't honouring the (8192)
constraint.

So, as I don't at all understand the overall design of this stack, in
the best tradition of modern medicine, my "cure" is to treat the
symptoms (duh!)...

In the read() routine (around line 900 in __init__.py) in the event
there is more than the requested data, instead of running straight
into the assert, I write out the requested amount (typically 8192),
and stash the rest away in a variable "self._datastash", which I
create on the fly (I said it was nasty!)  On entering read(), before
fetching any new data, I check if there is anything in
self._datastash, and if so, I return it....

I'm sure that if I came understand exactly how the buffers in that
routine are properly being used, I could forge a more elegant
solution, but my head hurts too much already...

<snip somewhere_around_line=900, comments=mostly_removed>

            if size < 0:
                self._rbuf = StringIO.StringIO()  # reset _rbuf.  we
consume it via buf.
                while True:
                    ### start fixup
                    try:
                        if self._datastash:
                            data = self._datastash
                            self._datastash = ''
                        else:
                            data = self.recv(rbufsize)
                    except:
                        self._datastash = ''
                        ### end fixup (plus indented next line)
                        data = self.recv(rbufsize)
                    if not data:
                        break
                    buf.write(data)
                return buf.getvalue()
            else:
<snippy />
                while True:
                    left = size - buf_len
                    ### start fixup
                    try:
                        if self._datastash:
                            data = self._datastash
                            self._datastash = ''
                        else:
                            data = self.recv( left)
                    except:
                        self._datastash = ''
                        ### end fixup (plus indented next line)
                        data = self.recv( left)
                    if not data:
                        break
                    n = len(data)
                    if n == size and not buf_len:
                        return data
                    if n == left:
                        buf.write(data)
                        del data  # explicit free
                        break
                    ### start fixup
                    if n > left:
                        buf.write(data[ :left])
                        self._datastash = data[ left:]
                        del data  # explicit free
                        break
                    ### end fixup
                    assert n <= left, "recv(%d) returned %d bytes" %
(left, n)
                    buf.write(data)
                    buf_len += n
                    del data  # explicit free
                    #assert buf_len == buf.tell()
                return buf.getvalue()
</snip>

I hope this is useful to someone, with luck, to sort it out properly
(somehow) for the next release...
Thanks, Alan

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.

Tim
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS and file uploads (a bug? Or is it just me?)

Tim
A fix for this has been committed see https://bitbucket.org/stpierre/cherrypy/changeset/1001a34c0fd52f91353b4b8e66ff444026f95ed5#chg-cherrypy/wsgiserver/wsgiserver2.py I replaced my wsgiserver2.py and my uploads are working fine with https now.

On Wednesday, October 27, 2010 9:37:37 AM UTC-5, alanp wrote:
So I have a "work-around" (aka a really horrible cruft!)

The first observation is that someone put the assert in the read()
function "__init__.py" to detect precisely this problematic condition
- that the recv( 8192) call returns more than 8192... And yes, under
certain conditions, under ssl, it clearly does (but maybe not for
everybody!)

I put stack trace into the recv function and tested uploading a 10k
file, and it is clear that when read() calls self.recv(8192), the call
initially visits a recv() fn in the ssl library, which then calls down
to the __init__ recv() function *twice*, first getting 8192 bytes,
then getting a further 2k, all of which (10k) it then returns to the
read function.

So it seems to be the ssl library that isn't honouring the (8192)
constraint.

So, as I don't at all understand the overall design of this stack, in
the best tradition of modern medicine, my "cure" is to treat the
symptoms (duh!)...

In the read() routine (around line 900 in __init__.py) in the event
there is more than the requested data, instead of running straight
into the assert, I write out the requested amount (typically 8192),
and stash the rest away in a variable "self._datastash", which I
create on the fly (I said it was nasty!)  On entering read(), before
fetching any new data, I check if there is anything in
self._datastash, and if so, I return it....

I'm sure that if I came understand exactly how the buffers in that
routine are properly being used, I could forge a more elegant
solution, but my head hurts too much already...

<snip somewhere_around_line=900, comments=mostly_removed>

            if size < 0:
                self._rbuf = StringIO.StringIO()  # reset _rbuf.  we
consume it via buf.
                while True:
                    ### start fixup
                    try:
                        if self._datastash:
                            data = self._datastash
                            self._datastash = ''
                        else:
                            data = self.recv(rbufsize)
                    except:
                        self._datastash = ''
                        ### end fixup (plus indented next line)
                        data = self.recv(rbufsize)
                    if not data:
                        break
                    buf.write(data)
                return buf.getvalue()
            else:
<snippy />
                while True:
                    left = size - buf_len
                    ### start fixup
                    try:
                        if self._datastash:
                            data = self._datastash
                            self._datastash = ''
                        else:
                            data = self.recv( left)
                    except:
                        self._datastash = ''
                        ### end fixup (plus indented next line)
                        data = self.recv( left)
                    if not data:
                        break
                    n = len(data)
                    if n == size and not buf_len:
                        return data
                    if n == left:
                        buf.write(data)
                        del data  # explicit free
                        break
                    ### start fixup
                    if n > left:
                        buf.write(data[ :left])
                        self._datastash = data[ left:]
                        del data  # explicit free
                        break
                    ### end fixup
                    assert n <= left, "recv(%d) returned %d bytes" %
(left, n)
                    buf.write(data)
                    buf_len += n
                    del data  # explicit free
                    #assert buf_len == buf.tell()
                return buf.getvalue()
</snip>

I hope this is useful to someone, with luck, to sort it out properly
(somehow) for the next release...
Thanks, Alan

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cherrypy-users/-/vtZ1aoKDxpAJ.
To post to this group, send email to [hidden email].
To unsubscribe from this group, send email to [hidden email].
For more options, visit this group at http://groups.google.com/group/cherrypy-users?hl=en.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS and file uploads (a bug? Or is it just me?)

Stephan Semerad
You guys are my heros! :) thanks a lot

On Wednesday, November 21, 2012 at 9:10:25 PM UTC+1, Tim wrote:
A fix for this has been committed see <a href="https://bitbucket.org/stpierre/cherrypy/changeset/1001a34c0fd52f91353b4b8e66ff444026f95ed5#chg-cherrypy/wsgiserver/wsgiserver2.py" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fstpierre%2Fcherrypy%2Fchangeset%2F1001a34c0fd52f91353b4b8e66ff444026f95ed5%23chg-cherrypy%2Fwsgiserver%2Fwsgiserver2.py\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8GJKb5q_SyAhafTRZ7akyr5xAOg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fbitbucket.org%2Fstpierre%2Fcherrypy%2Fchangeset%2F1001a34c0fd52f91353b4b8e66ff444026f95ed5%23chg-cherrypy%2Fwsgiserver%2Fwsgiserver2.py\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG8GJKb5q_SyAhafTRZ7akyr5xAOg&#39;;return true;">https://bitbucket.org/stpierre/cherrypy/changeset/1001a34c0fd52f91353b4b8e66ff444026f95ed5#chg-cherrypy/wsgiserver/wsgiserver2.py I replaced my wsgiserver2.py and my uploads are working fine with https now.

On Wednesday, October 27, 2010 9:37:37 AM UTC-5, alanp wrote:
So I have a "work-around" (aka a really horrible cruft!)

The first observation is that someone put the assert in the read()
function "__init__.py" to detect precisely this problematic condition
- that the recv( 8192) call returns more than 8192... And yes, under
certain conditions, under ssl, it clearly does (but maybe not for
everybody!)

I put stack trace into the recv function and tested uploading a 10k
file, and it is clear that when read() calls self.recv(8192), the call
initially visits a recv() fn in the ssl library, which then calls down
to the __init__ recv() function *twice*, first getting 8192 bytes,
then getting a further 2k, all of which (10k) it then returns to the
read function.

So it seems to be the ssl library that isn't honouring the (8192)
constraint.

So, as I don't at all understand the overall design of this stack, in
the best tradition of modern medicine, my "cure" is to treat the
symptoms (duh!)...

In the read() routine (around line 900 in __init__.py) in the event
there is more than the requested data, instead of running straight
into the assert, I write out the requested amount (typically 8192),
and stash the rest away in a variable "self._datastash", which I
create on the fly (I said it was nasty!)  On entering read(), before
fetching any new data, I check if there is anything in
self._datastash, and if so, I return it....

I'm sure that if I came understand exactly how the buffers in that
routine are properly being used, I could forge a more elegant
solution, but my head hurts too much already...

<snip somewhere_around_line=900, comments=mostly_removed>

            if size < 0:
                self._rbuf = StringIO.StringIO()  # reset _rbuf.  we
consume it via buf.
                while True:
                    ### start fixup
                    try:
                        if self._datastash:
                            data = self._datastash
                            self._datastash = ''
                        else:
                            data = self.recv(rbufsize)
                    except:
                        self._datastash = ''
                        ### end fixup (plus indented next line)
                        data = self.recv(rbufsize)
                    if not data:
                        break
                    buf.write(data)
                return buf.getvalue()
            else:
<snippy />
                while True:
                    left = size - buf_len
                    ### start fixup
                    try:
                        if self._datastash:
                            data = self._datastash
                            self._datastash = ''
                        else:
                            data = self.recv( left)
                    except:
                        self._datastash = ''
                        ### end fixup (plus indented next line)
                        data = self.recv( left)
                    if not data:
                        break
                    n = len(data)
                    if n == size and not buf_len:
                        return data
                    if n == left:
                        buf.write(data)
                        del data  # explicit free
                        break
                    ### start fixup
                    if n > left:
                        buf.write(data[ :left])
                        self._datastash = data[ left:]
                        del data  # explicit free
                        break
                    ### end fixup
                    assert n <= left, "recv(%d) returned %d bytes" %
(left, n)
                    buf.write(data)
                    buf_len += n
                    del data  # explicit free
                    #assert buf_len == buf.tell()
                return buf.getvalue()
</snip>

I hope this is useful to someone, with luck, to sort it out properly
(somehow) for the next release...
Thanks, Alan

--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/d/optout.
Loading...