How to quickly read a Z-axis cut of a multipage TIFF images?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to quickly read a Z-axis cut of a multipage TIFF images?

Yury V. Zaytsev
Hello PIL experts,

I am writing a Python script to analyse a multipage TIFF image stacks
coming from a microscope. The average file size is around several
gigabytes with few thousand grayscale frames and the resolution is
around 640x480 or thereabouts.

For this I need to apply some temporal filtering to individual pixels
(temporal means that I need to construct a "line" from the values of
this pixel along the pages).

Currently I am using Image.getpixel() in cycle, but it turns out that
it's prohibitively slow (much slower than the filtering that I am
applying). I estimated that on my 2 Ghz laptop it will take around 400
hours to analyse one pack (all the pixels).

Therefore I'd like to inquire whether this is an issue with PIL, or
underlying TIFF reader, or maybe there's a built-in method to quickly
read Z-axis cut that I didn't consider (I'm a total PIL n00b).

Also I was considering that maybe reading the whole page with some
built-in method, such as Image.getdata() and switching through the pages
may be faster, but I can't store the whole stack in memory (the stacks
that we have now will take around ~4 Gb, but this will quickly grow...).

Fortunately I can apply filters to a window, but this will introduce a
lot of complication in the algorithm, so I'm really looking forward to
hearing from you whether it is possible to speed up my code as is.

Many thanks!

P.S. Snippet attached:

           import numpy as np

           def getvalue(im, position):

                pixel = im.getpixel( position )

                if isinstance(pixel, int):
                    return pixel
                else:
                    return np.sqrt( np.sum( np.array(pixel) ** 2 ) )

            im_name = 'bleaching.tiff'
            im = Image.open(im_name)

            page = 0
            line = []
            try:
                while True:
                    im.seek(page)
                    line.append(getvalue(im, (x, y) ))
                    page += 1
            except EOFError:
                pass

            <do something with line and take another pixel>
 
--
Sincerely yours,
Yury V. Zaytsev

_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: How to quickly read a Z-axis cut of a multipage TIFF images?

Yury V. Zaytsev
Hi!

On Mon, 2010-04-12 at 10:03 +0200, Yury V. Zaytsev wrote:

> Therefore I'd like to inquire whether this is an issue with PIL, or
> underlying TIFF reader, or maybe there's a built-in method to quickly
> read Z-axis cut that I didn't consider (I'm a total PIL n00b).

As a follow-up I have got another idea (which I would be quite reluctant
to implement though) on how to dramatically speed up things, that is to
convert the image to an uncompressed binary format which would allow
random seeking (given that the resolution is known and so is the number
of bytes per pixel).

Or even consider storing lines of pixel luminosity over time one after
another which of course would require to rewrite the whole file to
append frames, but this is not really an issue since the length of the
recording does not change.

However, I'd really like to hear to some other ideas, since these TIFF
files do not seem to be using any compression anyway, so in theory,
sequential seeking operation should not be too expensive?
 
--
Sincerely yours,
Yury V. Zaytsev

_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: How to quickly read a Z-axis cut of a multipage TIFF images?

Chris Barker
Yury V. Zaytsev wrote:
>> Therefore I'd like to inquire whether this is an issue with PIL, or
>> underlying TIFF reader, or maybe there's a built-in method to quickly
>> read Z-axis cut that I didn't consider (I'm a total PIL n00b).

I"m not sure I quite follow what you need to do, but in general, doing
things one pixel, or line, or... at a time is going to be slow in
Python. You can't hold it all in memory, but maybe you could work with a
bunch of frames at a time -- as many as you can hold in memory?

> As a follow-up I have got another idea (which I would be quite reluctant
> to implement though) on how to dramatically speed up things, that is to
> convert the image to an uncompressed binary format which would allow
> random seeking (given that the resolution is known and so is the number
> of bytes per pixel).

Maybe you could load it all into a numpy memory mapped array, and then
work with it there? Or maybe hdf5, via PyTables or H5Py.

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: How to quickly read a Z-axis cut of a multipage TIFF images?

Joao S. O. Bueno Calligaris
Hi Christopher -

Image these sizes are certainly not apropriate to be dealt with straight from Python code on a pixel by pixel basis. 

I once implementd a Tiff reader in C using libtiff, and it was not hard to do - maybe you should try and move part of your processing to C , and implement some Python itnerface that would allow you to drive everything with simpler scripting. 

(For example, you could have a libtiff C function that would create you an Y-Z slice of your data, and return that as a 2d array to python)


I don't know the hdf5 tables or H5Py Dr. barker mentioned above, but if you feel like purisuing the C-libtiff with python bindings line, feel free to contact me.

 
  js
  -><-


_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: How to quickly read a Z-axis cut of a multipage TIFF images?

Chris Barker
Joao S. O. Bueno wrote:
> I once implementd a Tiff reader in C using libtiff, and it was not hard
> to do - maybe you should try and move part of your processing to C , and
> implement some Python itnerface that would allow you to drive everything
> with simpler scripting.
>
> (For example, you could have a libtiff C function that would create you
> an Y-Z slice of your data, and return that as a 2d array to python)

If you go this route, I'd strongly recommend using Cython, or maybe
ctypes for the bindings.

-Chris



--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig
Reply | Threaded
Open this post in threaded view
|

Re: How to quickly read a Z-axis cut of a multipage TIFF images?

Sebastian Haase-3
Joao,

could you (shortly) outline what kind of  functions you could benefit
from using libtiff directly, rather than PIL (which uses raw python
for this, as
I understand).
Maybe you even show us a short example.
Another reason why I ask, is that I read that libtiff has functions
using memory mapping. So far, I have considered TIF a rather
inappropriate format for large (multi-dimensional) data because it's
not compatible to the mem-map operations I can do using numpy. (i.e. I
always have to read everything  sequentially from the start)

Thanks,
Sebastian Haase



On Mon, Apr 12, 2010 at 8:09 PM, Christopher Barker
<[hidden email]> wrote:

> Joao S. O. Bueno wrote:
>>
>> I once implementd a Tiff reader in C using libtiff, and it was not hard to
>> do - maybe you should try and move part of your processing to C , and
>> implement some Python itnerface that would allow you to drive everything
>> with simpler scripting.
>> (For example, you could have a libtiff C function that would create you an
>> Y-Z slice of your data, and return that as a 2d array to python)
>
> If you go this route, I'd strongly recommend using Cython, or maybe ctypes
> for the bindings.
>
> -Chris
>
>
>
> --
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> [hidden email]
> _______________________________________________
> Image-SIG maillist  -  [hidden email]
> http://mail.python.org/mailman/listinfo/image-sig
>
_______________________________________________
Image-SIG maillist  -  [hidden email]
http://mail.python.org/mailman/listinfo/image-sig