[issue10376] ZipFile unzip is unbuffered

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor

New submission from James Hutchison <[hidden email]>:

The Unzip module is always unbuffered (tested v.3.1.2 Windows XP, 32-bit). This means that if one has to do many small reads it is a lot slower than reading a chunk of data to a buffer and then reading from that buffer. It seems logical that the unzip module should default to buffered reading and/or have a buffered argument. Likewise, the documentation should clarify that there is no buffering involved when doing a read, which runs contrary to the default behavior of a normal read.

start Zipfile read
done
27432 reads done
took 0.859 seconds
start buffered Zipfile read
done
27432 reads done
took 0.072 seconds
start normal read (default buffer)
done
27432 reads done
took 0.139 seconds
start buffered normal read
done
27432
took 0.137 seconds

----------
assignee: docs@python
components: Documentation, IO, Library (Lib)
messages: 120871
nosy: Jimbofbx, docs@python
priority: normal
severity: normal
status: open
title: ZipFile unzip is unbuffered
type: performance
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor

James Hutchison <[hidden email]> added the comment:

I should clarify that this is the zipfile constructor I am using:

zipfile.ZipFile(filename, mode='r', allowZip64=True);

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor
In reply to this post by STINNER Victor

Changes by Terry J. Reedy <[hidden email]>:


----------
versions: +Python 3.2, Python 3.3 -Python 2.5, Python 2.6, Python 3.1

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor
In reply to this post by STINNER Victor

Changes by Xuanji Li <[hidden email]>:


----------
nosy: +xuanji

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor
In reply to this post by STINNER Victor

Changes by Serhiy Storchaka <[hidden email]>:


----------
nosy: +storchaka

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor
In reply to this post by STINNER Victor

Serhiy Storchaka <[hidden email]> added the comment:

Actually reading from the zip file is buffered (at least 4 KiB of uncompressed data at a time).

Can you give tests, scripts and data, which show the problem?

----------

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com

Reply | Threaded
Open this post in threaded view
|

[issue10376] ZipFile unzip is unbuffered

STINNER Victor
In reply to this post by STINNER Victor

James Hutchison <[hidden email]> added the comment:

See attached, which will open a zipfile that contains one file and reads it a bunch of times using unbuffered and buffered idioms. This was tested on windows using python 3.2

You're in charge of coming up with a file to test it on. Sorry.

Example output:

Enter filename: test.zip
Timing unbuffered read, 5 bytes at a time. 10 loops
took 6.671999931335449
Timing buffered read, 5 bytes at a time (4000 byte buffer). 10 loops
took 0.7350001335144043

----------
Added file: http://bugs.python.org/file25432/zipfiletest.py

_______________________________________
Python tracker <[hidden email]>
<http://bugs.python.org/issue10376>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%2B1322467933539-512619%40n6.nabble.com