Posting gzip'd image file - server says Malformed Upload?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Paul Hubert
Any idea why a server might be returning the message, in json format, "Malformed Upload"? The image is gzipped as the server requires... Someone on another forum said that he thinks Python requests automatically gzips?

I have the code working in vb.net... no idea why it wont work in Python.

This is my code.

def uploadPhoto(url, sid, dafile):

headers = {'User-Agent':useragent,
                     'Session-Id':sessionid,
                     'Content-Type':'image/jpeg',
                     'Accept-Encoding':'gzip, deflate'}

f_in = open(dafile, 'rb')
f_out = gzip.open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()

files = {('Image', open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'rb'))}
r = requests.post(url, headers=headers, proxies=proxies, files=files)    

print r.status_code
print r.request.headers
return str(r.text)



The server replies with:

{"message":"Malformed Upload","code":1012}

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Chris Angelico
On Thu, Jun 18, 2015 at 7:55 AM, Paul Hubert <phbrt25 at gmail.com> wrote:
> f_in = open(dafile, 'rb')
> f_out = gzip.open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'wb')
> f_out.writelines(f_in)
> f_out.close()
> f_in.close()

Are you sure you want iteration and writelines() here? I would be
inclined to avoid those for any situation that isn't plain text. If
the file isn't too big, I'd just read it all in a single blob and then
write it all out at once.

ChrisA

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Paul Hubert
In reply to this post by Paul Hubert
On Wednesday, June 17, 2015 at 8:24:17 PM UTC-4, Chris Angelico wrote:

> Are you sure you want iteration and writelines() here? I would be
> inclined to avoid those for any situation that isn't plain text. If
> the file isn't too big, I'd just read it all in a single blob and then
> write it all out at once.
>
> ChrisA

Do you think that would fix my issue? Could you give me an example?

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Michael Torrie
On 06/17/2015 06:45 PM, Paul Hubert wrote:

> On Wednesday, June 17, 2015 at 8:24:17 PM UTC-4, Chris Angelico wrote:
>
>> Are you sure you want iteration and writelines() here? I would be
>> inclined to avoid those for any situation that isn't plain text. If
>> the file isn't too big, I'd just read it all in a single blob and then
>> write it all out at once.
>>
>> ChrisA
>
> Do you think that would fix my issue?

I recommend you try it and see.  That's the only way to know.  We don't
know any details of your program or situation really, so we can only guess.

> Could you give me an example?

Instead of writelines(), just write() the data.  There are examples in
the official python docs I'm sure.

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Chris Angelico
In reply to this post by Paul Hubert
On Thu, Jun 18, 2015 at 10:45 AM, Paul Hubert <phbrt25 at gmail.com> wrote:

> On Wednesday, June 17, 2015 at 8:24:17 PM UTC-4, Chris Angelico wrote:
>
>> Are you sure you want iteration and writelines() here? I would be
>> inclined to avoid those for any situation that isn't plain text. If
>> the file isn't too big, I'd just read it all in a single blob and then
>> write it all out at once.
>>
>> ChrisA
>
> Do you think that would fix my issue? Could you give me an example?

Sorry for the abrupt and terse previous email; I had a student arrive
just as I was posting that, and hit Ctrl-Enter when I should really
have just left the email as a draft. Here's what I'm thinking:

# Was:
f_in = open(dafile, 'rb')
f_out = gzip.open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()

# Now:
gz = '/Users/Paul/Desktop/scripts/pic.jpg.gz'
with open(dafile, 'rb') as f_in, gzip.open(gz, 'wb') as f_out:
    f_out.write(f_in.read())

You might actually be able to write to a StringIO rather than to a
file, given that you appear to be just reading the data back again
straight away. But in case you want to keep the file around for some
other reason, this still works the exact same way you had it.

The main difference is that this version swallows the entire file in a
single gulp, then passes it all to the gzip writer. If your file is
tiny (under 1MB), this is perfect. If it's huge (over 1GB), you may
have problems. In between, it'll probably work, but might be
inefficient.

Hope that helps!

ChrisA

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Paul Hubert
In reply to this post by Paul Hubert
On Wednesday, June 17, 2015 at 9:46:25 PM UTC-4, Chris Angelico wrote:

> On Thu, Jun 18, 2015 at 10:45 AM, Paul Hubert <phbrt25 at gmail.com> wrote:
> > On Wednesday, June 17, 2015 at 8:24:17 PM UTC-4, Chris Angelico wrote:
> >
> >> Are you sure you want iteration and writelines() here? I would be
> >> inclined to avoid those for any situation that isn't plain text. If
> >> the file isn't too big, I'd just read it all in a single blob and then
> >> write it all out at once.
> >>
> >> ChrisA
> >
> > Do you think that would fix my issue? Could you give me an example?
>
> Sorry for the abrupt and terse previous email; I had a student arrive
> just as I was posting that, and hit Ctrl-Enter when I should really
> have just left the email as a draft. Here's what I'm thinking:
>
> # Was:
> f_in = open(dafile, 'rb')
> f_out = gzip.open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'wb')
> f_out.writelines(f_in)
> f_out.close()
> f_in.close()
>
> # Now:
> gz = '/Users/Paul/Desktop/scripts/pic.jpg.gz'
> with open(dafile, 'rb') as f_in, gzip.open(gz, 'wb') as f_out:
>     f_out.write(f_in.read())
>
> You might actually be able to write to a StringIO rather than to a
> file, given that you appear to be just reading the data back again
> straight away. But in case you want to keep the file around for some
> other reason, this still works the exact same way you had it.
>
> The main difference is that this version swallows the entire file in a
> single gulp, then passes it all to the gzip writer. If your file is
> tiny (under 1MB), this is perfect. If it's huge (over 1GB), you may
> have problems. In between, it'll probably work, but might be
> inefficient.
>
> Hope that helps!
>
> ChrisA

Same result - server says malformed upload. :/

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Chris Angelico
On Thu, Jun 18, 2015 at 1:48 PM, Paul Hubert <phbrt25 at gmail.com> wrote:
>> # Now:
>> gz = '/Users/Paul/Desktop/scripts/pic.jpg.gz'
>> with open(dafile, 'rb') as f_in, gzip.open(gz, 'wb') as f_out:
>>     f_out.write(f_in.read())
>>
>
> Same result - server says malformed upload. :/

Oh well, was worth a shot. Since the file's sitting around still, you
should be able to attempt to decompress it with the stand-alone gzip
command; does that give the right result? Also, you should be able to
manually send that .gz file up to the server, which would be worth
confirming.

Are you certain that the content should be gzipped as a file? It's not
meant to be done as a transfer-level encoding?

ChrisA

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Michael Torrie
In reply to this post by Paul Hubert
On 06/17/2015 09:48 PM, Paul Hubert wrote:
> Same result - server says malformed upload. :/

You may want to run a sniffer like wireshark and see what the difference
is between the packets coming from your C# program and coming from Python.



Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

random832@fastmail.us
In reply to this post by Chris Angelico
On Wed, Jun 17, 2015, at 20:23, Chris Angelico wrote:

> On Thu, Jun 18, 2015 at 7:55 AM, Paul Hubert <phbrt25 at gmail.com> wrote:
> > f_in = open(dafile, 'rb')
> > f_out = gzip.open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'wb')
> > f_out.writelines(f_in)
> > f_out.close()
> > f_in.close()
>
> Are you sure you want iteration and writelines() here? I would be
> inclined to avoid those for any situation that isn't plain text. If
> the file isn't too big, I'd just read it all in a single blob and then
> write it all out at once.

Is there a reason not to use shutil.copyfileobj?

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Chris Angelico
On Thu, Jun 18, 2015 at 10:38 PM,  <random832 at fastmail.us> wrote:

> On Wed, Jun 17, 2015, at 20:23, Chris Angelico wrote:
>> On Thu, Jun 18, 2015 at 7:55 AM, Paul Hubert <phbrt25 at gmail.com> wrote:
>> > f_in = open(dafile, 'rb')
>> > f_out = gzip.open('/Users/Paul/Desktop/scripts/pic.jpg.gz', 'wb')
>> > f_out.writelines(f_in)
>> > f_out.close()
>> > f_in.close()
>>
>> Are you sure you want iteration and writelines() here? I would be
>> inclined to avoid those for any situation that isn't plain text. If
>> the file isn't too big, I'd just read it all in a single blob and then
>> write it all out at once.
>
> Is there a reason not to use shutil.copyfileobj?

If the file is too big (or might be too big) to want to fit into
memory, then sure. But a typical JPEG image isn't going to be
gigabytes of content; chances are it's going to be a meg or so at
most, and that doesn't justify the overhead of chunking. Just read it,
write it, job done.

ChrisA

Reply | Threaded
Open this post in threaded view
|

Posting gzip'd image file - server says Malformed Upload?

Laura Creighton-2
In reply to this post by Chris Angelico
I got to this party late.

One way to get the malformed upload message is is you gzip something
that already is gzipped, and send that up the pipe.

worth checking.

Laura