javascript / Python communication for image viewer

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

javascript / Python communication for image viewer

matthewbrett
Guys / gals,

I want to ask for advice about writing a brain image display widget for
IPython.

I would like to make an IPython widget that can take an in-memory numpy array
and do an interactive display of orthogonal slices from the array.  The
display will look something like Papaya:

http://rii.uthscsa.edu/mango/papaya

where clicking or moving the mouse causes matching slices to be displayed
through the three axes of the numpy array (here a brain image).

Papaya is pure javascript, so I am assuming that it loads the whole array
(brain image) into a javascript variable and takes slices from that.

What I would like to do, is to be able to keep the whole 3D array only in
Python, and pass the slices as needed to a javascript viewer.

In my ignorance, I am not sure which approach to go for first.

Should I use the comm / widget interface for this?  In that case I guess the
procedure would be:

* mouse movement generates a 'need slice' message from javascript
* python kernel accepts 'need slice' message, takes slice from array, base64
  encodes into JSON, sends 'here is your slice' message back to javascript
  with the data
* javascript accepts message, unpacks base64'ed JSONed slice into variable and
  displays slice variable

Is that right?  I guess that involves Is there any chance that this
will be fast enough for
satisfying interactive movement through the image, which would likely require
something like 20 slices per second, each of say 64 x 64 floating point?

If not - is there something else I should look at instead?

Another question for more thanks - should I use a Canvas or SVG element to
display the images?  The fabric.js README [1] seems to imply the
Canvas element is
faster for some interactive stuff, does anyone have relevant experience to
share?

Thanks a lot,

Matthew

[1] https://github.com/kangax/fabric.js/#history
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: javascript / Python communication for image viewer

rossant
We have pretty similar requirements for Vispy, as we target fast
visualization of big datasets using OpenGL in the IPython notebook. In
particular, we can't use Canvas or SVG because it's just too slow for
big data, so we have to use WebGL. This lets us leverage the GPU for
visualization.

I would be very interested in seeing the answers to your questions.
Notably, is it possible to use binary sockets for transferring large
amounts of binary data between Python and JavaScript?

Cyrille

2014-08-27 20:52 GMT+01:00 Matthew Brett <[hidden email]>:

> Guys / gals,
>
> I want to ask for advice about writing a brain image display widget for
> IPython.
>
> I would like to make an IPython widget that can take an in-memory numpy array
> and do an interactive display of orthogonal slices from the array.  The
> display will look something like Papaya:
>
> http://rii.uthscsa.edu/mango/papaya
>
> where clicking or moving the mouse causes matching slices to be displayed
> through the three axes of the numpy array (here a brain image).
>
> Papaya is pure javascript, so I am assuming that it loads the whole array
> (brain image) into a javascript variable and takes slices from that.
>
> What I would like to do, is to be able to keep the whole 3D array only in
> Python, and pass the slices as needed to a javascript viewer.
>
> In my ignorance, I am not sure which approach to go for first.
>
> Should I use the comm / widget interface for this?  In that case I guess the
> procedure would be:
>
> * mouse movement generates a 'need slice' message from javascript
> * python kernel accepts 'need slice' message, takes slice from array, base64
>   encodes into JSON, sends 'here is your slice' message back to javascript
>   with the data
> * javascript accepts message, unpacks base64'ed JSONed slice into variable and
>   displays slice variable
>
> Is that right?  I guess that involves Is there any chance that this
> will be fast enough for
> satisfying interactive movement through the image, which would likely require
> something like 20 slices per second, each of say 64 x 64 floating point?
>
> If not - is there something else I should look at instead?
>
> Another question for more thanks - should I use a Canvas or SVG element to
> display the images?  The fabric.js README [1] seems to imply the
> Canvas element is
> faster for some interactive stuff, does anyone have relevant experience to
> share?
>
> Thanks a lot,
>
> Matthew
>
> [1] https://github.com/kangax/fabric.js/#history
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: javascript / Python communication for image viewer

James Booth
Hey Matthew, Cyrille,

I’m developing a three.js-based web app for image/object annotation called landmarker.io (I work in computer vision/machine learning, getting high quality annotations of 3D and 2D data is often critical to training models). It’s located here

www.landmarker.io (tool, launches in demo by default)
https://github.com/menpo/landmarker.io (code)

I’m also one of the developer’s of Menpo (www.menpo.io), which is a Python package for building and testing deformable models. I’m going to be developing an IPython widget-version of the landmarker.io tool, so we can correct annotations as we sport problems in the notebook.

I mention all this as:

1. I’m also interested in understanding how we can transfer large arrays efficiently to IPython widgets
2. I have a little experience in how best to handle this in a traditional client/server model, but I don’t know how well this will translate to the comm interface.

Basically, landmarker.io expects to be able to talk to a RESTful interface that is implemented currently by:

https://github.com/menpo/landmarkerio-server

Originally, I sent meshes as large JSON objects, but this was pretty slow as:

1. It’s more work in JS to parse the JSON into an array
2. It’s more work to turn this into the most efficient JS type for numerical arrays (which WebGL loves) which is an ArrayBuffer.

https://developer.mozilla.org/en-US/docs/Web/API/ArrayBuffer
https://developer.mozilla.org/en-US/docs/Web/API/Float32Array

This meant that even with a server on localhost, skipping through faces to annotate felt a little sluggish.

Instead now I just directly build a pure ArrayBuffer in Python on the server and ship it to the client. That happens here in the server:

https://github.com/menpo/landmarkerio-server/blob/master/landmarkerio/cache.py#L165

(I’m a little lazy here and literally save the file to disk, then gzip it. It could be done in memory with StringIO through, but I do this as a caching process so it’s going to disk anyway).

On the client side, I make an XMLHttpRequest and demand an array buffer from the server:
https://github.com/menpo/landmarker.io/blob/master/src/js/app/lib/get.js#L9

For completion, the actual parsing of the array buffer is done here:
https://github.com/menpo/landmarker.io/blob/master/src/js/app/model/mesh.js#L104

Because three.js supports ArrayBuffers, this is really fast. I just point three to the array and we are away. With this implementation, browsing through subjects to annotate with a server on localhost (akin to moving between slices in your brain scan I imagine) is very fast.

I’m afraid I don’t have much knowledge of the comm interface - is there a document I could be pointed out that lays out the protocol? Does it sound possible to send a pure array to JS in the way I’m doing in the landmarkerio-server?

Best,
James



Sent from Mailbox


On Wed, Aug 27, 2014 at 9:53 PM, Cyrille Rossant <[hidden email]> wrote:

We have pretty similar requirements for Vispy, as we target fast
visualization of big datasets using OpenGL in the IPython notebook. In
particular, we can't use Canvas or SVG because it's just too slow for
big data, so we have to use WebGL. This lets us leverage the GPU for
visualization.

I would be very interested in seeing the answers to your questions.
Notably, is it possible to use binary sockets for transferring large
amounts of binary data between Python and JavaScript?

Cyrille

2014-08-27 20:52 GMT+01:00 Matthew Brett <[hidden email]>:


> Guys / gals,
>
> I want to ask for advice about writing a brain image display widget for
> IPython.
>
> I would like to make an IPython widget that can take an in-memory numpy array
> and do an interactive display of orthogonal slices from the array. The
> display will look something like Papaya:
>
> http://rii.uthscsa.edu/mango/papaya
>
> where clicking or moving the mouse causes matching slices to be displayed
> through the three axes of the numpy array (here a brain image).
>
> Papaya is pure javascript, so I am assuming that it loads the whole array
> (brain image) into a javascript variable and takes slices from that.
>
> What I would like to do, is to be able to keep the whole 3D array only in
> Python, and pass the slices as needed to a javascript viewer.
>
> In my ignorance, I am not sure which approach to go for first.
>
> Should I use the comm / widget interface for this? In that case I guess the
> procedure would be:
>
> * mouse movement generates a 'need slice' message from javascript
> * python kernel accepts 'need slice' message, takes slice from array, base64
> encodes into JSON, sends 'here is your slice' message back to javascript
> with the data
> * javascript accepts message, unpacks base64'ed JSONed slice into variable and
> displays slice variable
>
> Is that right? I guess that involves Is there any chance that this
> will be fast enough for
> satisfying interactive movement through the image, which would likely require
> something like 20 slices per second, each of say 64 x 64 floating point?
>
> If not - is there something else I should look at instead?
>
> Another question for more thanks - should I use a Canvas or SVG element to
> display the images? The fabric.js README [1] seems to imply the
> Canvas element is
> faster for some interactive stuff, does anyone have relevant experience to
> share?
>
> Thanks a lot,
>
> Matthew
>
> [1] https://github.com/kangax/fabric.js/#history
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev



_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev
Reply | Threaded
Open this post in threaded view
|

Re: javascript / Python communication for image viewer

Matthias Bussonnier
Hi all, 


You probably don’t want to use widget framework directly but more comms object. 
Binary data exchange will be possible with https://github.com/ipython/ipython/pull/6110

— 
M
Le 28 août 2014 à 08:45, James Booth <[hidden email]> a écrit :

Hey Matthew, Cyrille,

I’m developing a three.js-based web app for image/object annotation called landmarker.io (I work in computer vision/machine learning, getting high quality annotations of 3D and 2D data is often critical to training models). It’s located here

www.landmarker.io (tool, launches in demo by default)

I’m also one of the developer’s of Menpo (www.menpo.io), which is a Python package for building and testing deformable models. I’m going to be developing an IPython widget-version of the landmarker.io tool, so we can correct annotations as we sport problems in the notebook.

I mention all this as:

1. I’m also interested in understanding how we can transfer large arrays efficiently to IPython widgets
2. I have a little experience in how best to handle this in a traditional client/server model, but I don’t know how well this will translate to the comm interface.

Basically, landmarker.io expects to be able to talk to a RESTful interface that is implemented currently by:


Originally, I sent meshes as large JSON objects, but this was pretty slow as:

1. It’s more work in JS to parse the JSON into an array
2. It’s more work to turn this into the most efficient JS type for numerical arrays (which WebGL loves) which is an ArrayBuffer.


This meant that even with a server on localhost, skipping through faces to annotate felt a little sluggish.

Instead now I just directly build a pure ArrayBuffer in Python on the server and ship it to the client. That happens here in the server:

(I’m a little lazy here and literally save the file to disk, then gzip it. It could be done in memory with StringIO through, but I do this as a caching process so it’s going to disk anyway).

On the client side, I make an XMLHttpRequest and demand an array buffer from the server:
For completion, the actual parsing of the array buffer is done here:
Because three.js supports ArrayBuffers, this is really fast. I just point three to the array and we are away. With this implementation, browsing through subjects to annotate with a server on localhost (akin to moving between slices in your brain scan I imagine) is very fast.

I’m afraid I don’t have much knowledge of the comm interface - is there a document I could be pointed out that lays out the protocol? Does it sound possible to send a pure array to JS in the way I’m doing in the landmarkerio-server?

Best,
James



Sent from Mailbox


On Wed, Aug 27, 2014 at 9:53 PM, Cyrille Rossant <[hidden email]> wrote:

We have pretty similar requirements for Vispy, as we target fast
visualization of big datasets using OpenGL in the IPython notebook. In
particular, we can't use Canvas or SVG because it's just too slow for
big data, so we have to use WebGL. This lets us leverage the GPU for
visualization.

I would be very interested in seeing the answers to your questions.
Notably, is it possible to use binary sockets for transferring large
amounts of binary data between Python and JavaScript?

Cyrille

2014-08-27 20:52 GMT+01:00 Matthew Brett <[hidden email]>:


> Guys / gals,
>
> I want to ask for advice about writing a brain image display widget for
> IPython.
>
> I would like to make an IPython widget that can take an in-memory numpy array
> and do an interactive display of orthogonal slices from the array. The
> display will look something like Papaya:
>
> http://rii.uthscsa.edu/mango/papaya
>
> where clicking or moving the mouse causes matching slices to be displayed
> through the three axes of the numpy array (here a brain image).
>
> Papaya is pure javascript, so I am assuming that it loads the whole array
> (brain image) into a javascript variable and takes slices from that.
>
> What I would like to do, is to be able to keep the whole 3D array only in
> Python, and pass the slices as needed to a javascript viewer.
>
> In my ignorance, I am not sure which approach to go for first.
>
> Should I use the comm / widget interface for this? In that case I guess the
> procedure would be:
>
> * mouse movement generates a 'need slice' message from javascript
> * python kernel accepts 'need slice' message, takes slice from array, base64
> encodes into JSON, sends 'here is your slice' message back to javascript
> with the data
> * javascript accepts message, unpacks base64'ed JSONed slice into variable and
> displays slice variable
>
> Is that right? I guess that involves Is there any chance that this
> will be fast enough for
> satisfying interactive movement through the image, which would likely require
> something like 20 slices per second, each of say 64 x 64 floating point?
>
> If not - is there something else I should look at instead?
>
> Another question for more thanks - should I use a Canvas or SVG element to
> display the images? The fabric.js README [1] seems to imply the
> Canvas element is
> faster for some interactive stuff, does anyone have relevant experience to
> share?
>
> Thanks a lot,
>
> Matthew
>
> [1] https://github.com/kangax/fabric.js/#history
> _______________________________________________
> IPython-dev mailing list
> [hidden email]
> http://mail.scipy.org/mailman/listinfo/ipython-dev
_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev


_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev


_______________________________________________
IPython-dev mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-dev