bcolz, a column store for Python, 1.1.2 released

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

bcolz, a column store for Python, 1.1.2 released

Francesc Alted-4
Announcing bcolz 1.1.2

What's new

This is a maintenance release that brings quite a lot of improvements.
Here are the highlights:

- Zstd is a supported codec now.  Fixes #331.

- C-Blosc updated to 1.11.2.

- Added a new `defaults_ctx` context so that users can select defaults
  easily without changing global behaviour. For example::

   with bcolz.defaults_ctx(vm="python", cparams=bcolz.cparams(clevel=0)):
      cout = bcolz.eval("(x + 1) < 0")

- Fixed a crash occurring in `ctable.todataframe()` when both `columns`
  and `orient='columns'` were specified.  PR #311.  Thanks to Peter

- Use `pkg_resources.parse_version()` to test for version of packages.
  Fixes #322 (PY27 bcolz with dask unicode error).

- New package recipe for conda-forge.  Now you can install bcolz with:
`conda install -c conda-forge bcolz`.  Thanks to Alistair Miles.

For a more detailed change log, see:


For some comparison between bcolz and other compressed data containers,


specially chapters 3 (in-memory containers) and 4 (on-disk containers).

What it is

*bcolz* provides **columnar and compressed** data containers that can
live either on-disk or in-memory.  The compression is carried out
transparently by Blosc, an ultra fast meta-compressor that is optimized
for binary data.  Compression is active by default.

Column storage allows for efficiently querying tables with a large
number of columns.  It also allows for cheap addition and removal of
columns.  Lastly, high-performance iterators (like ``iter()``,
``where()``) for querying the objects are provided.

bcolz can use diffent backends internally (currently numexpr,
Python/NumPy or dask) so as to accelerate many vector and query
operations (although it can use pure NumPy for doing so too).  Moreover,
since the carray/ctable containers can be disk-based, it is possible to
use them for seamlessly performing out-of-memory computations.

While NumPy is used as the standard way to feed and retrieve data from
bcolz internal containers, but it also comes with support for
high-performance import/export facilities to/from `HDF5/PyTables tables
<http://www.pytables.org>`_ and `pandas dataframes

Have a look at how bcolz and the Blosc compressor, are making a better
use of the memory without an important overhead, at least for some real


bcolz has minimal dependencies (NumPy is the only strict requisite),
comes with an exhaustive test suite, and it is meant to be used in
production. Example users of bcolz are Visualfabriq
(http://www.visualfabriq.com/), Quantopian (https://www.quantopian.com/)
and scikit-allel:

* Visualfabriq:

  * *bquery*, A query and aggregation framework for Bcolz:
  * https://github.com/visualfabriq/bquery

* Quantopian:

  * Using compressed data containers for faster backtesting at scale:
  * https://quantopian.github.io/talks/NeedForSpeed/slides.html

* scikit-allel:

  * Exploratory analysis of large scale genetic variation data.
  * https://github.com/cggh/scikit-allel


Visit the main bcolz site repository at:


Home of Blosc compressor:

User's mail list:
[hidden email]

License is the new BSD:

Release notes can be found in the Git repository:


  **Enjoy data!**

Francesc Alted

        Support the Python Software Foundation: