[IPython-User] using iPython as a way to generate repetitive reports.

[Zachary Charlop-Powers]

I recently posted a question to Stack Overflow about the possibility of using iPython to generate reproducible, repetitive reports. (http://stackoverflow.com/questions/13614783/programatically-add-cells-to-an-ipython-notebook-for-report-generation) Lets say that I have a number of Pandas data frames and for each data frame I want to generate a series of tables , graphs and text, and then combine these into a report. I could use a templating system like Jinja but I am curious to know if I could use the iPython framework of ipython-notebook ---> nbconverter ---> pdf/html/etc for the purpose of generating all of the data analyses and the report into a single document. I have followed Thomas K.'s suggestion to use the get_ipython().set_next_input() function to add text to the next code block. For my idea to work, a few things would need to work:

example code.

from IPython.core.display import Image, display_png

x = Image(filename='test.png')

for i in range(3):

x = Image(filename='/Users/zachpowers/Downloads/pics/Sieve_1.png') 

 get_ipython().set_next_input('Thats What I am Talking About!') 

 get_ipython().set_next_input(display_png(x))

if you run this you will see that there are a few different problems that you encounter.

1) you can't create Markdown cells  

2) the display_png() function does not respect the .set_next_input() function so they are all displayed in the same output block in which they are called

3) the .set_next_input() function would need to be modified to track which code blocks it insert.  It iterates over the loop and inserts a code block below the code block it sits in with the result that the code blocks are added in reverse order from expected.

4) there would need to be a way to hide the code-block that calls the rest of the functions.

5) you would probably still need a templating engine to generate interesting text of each markdown block

So in closing I think the python ---> nbconvert is a great tool and that it might potentially be useful for reproducible research of repetitive reports (say that five times fast). I think that a few tweaks to the ipython notebook might make this possible and that the benefit would be the creation of a single file from which analyses could be rerun and from which reports could be generated. However, upon writing up my thoughts I am not sure that this is easier than doing the analysis in one file and outputting the results via a separate templating system. 

I hope some of the iPython community might find this line of thought to be useful and if so we could try to implement some of these ideas.

thanks,

zach cp

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

[Robert McGibbon]

Hey,

I think this is a very cool idea. I work with long-running (weeks to months) simulations, and we always want to know things like how converged our data collection is. Manually, we might load up the current data, make a "report" like some PCA biplots or what have you, and then decide from there whether to stop data collection or not. Being able to do this type of thing from a script (on a crontab) with the result being a formatted PDF would be awesome.

As you point out maybe a true templating system like jinja would be more appropriate, but for simple tasks (the reports don't have to be beautiful, they just need to contain some plots and text) this could be quite useful.

-Robert

On Dec 1, 2012, at 3:17 PM, Zachary Charlop-Powers wrote:

I recently posted a question to Stack Overflow about the possibility of using iPython to generate reproducible, repetitive reports. (http://stackoverflow.com/questions/13614783/programatically-add-cells-to-an-ipython-notebook-for-report-generation) Lets say that I have a number of Pandas data frames and for each data frame I want to generate a series of tables , graphs and text, and then combine these into a report. I could use a templating system like Jinja but I am curious to know if I could use the iPython framework of ipython-notebook ---> nbconverter ---> pdf/html/etc for the purpose of generating all of the data analyses and the report into a single document. I have followed Thomas K.'s suggestion to use the get_ipython().set_next_input() function to add text to the next code block. For my idea to work, a few things would need to work:

example code.

from IPython.core.display import Image, display_png

x = Image(filename='test.png')

for i in range(3):

x = Image(filename='/Users/zachpowers/Downloads/pics/Sieve_1.png') 

 get_ipython().set_next_input('Thats What I am Talking About!') 

 get_ipython().set_next_input(display_png(x))

if you run this you will see that there are a few different problems that you encounter.

1) you can't create Markdown cells  

2) the display_png() function does not respect the .set_next_input() function so they are all displayed in the same output block in which they are called

3) the .set_next_input() function would need to be modified to track which code blocks it insert.  It iterates over the loop and inserts a code block below the code block it sits in with the result that the code blocks are added in reverse order from expected.

4) there would need to be a way to hide the code-block that calls the rest of the functions.

5) you would probably still need a templating engine to generate interesting text of each markdown block

So in closing I think the python ---> nbconvert is a great tool and that it might potentially be useful for reproducible research of repetitive reports (say that five times fast). I think that a few tweaks to the ipython notebook might make this possible and that the benefit would be the creation of a single file from which analyses could be rerun and from which reports could be generated. However, upon writing up my thoughts I am not sure that this is easier than doing the analysis in one file and outputting the results via a separate templating system. 

I hope some of the iPython community might find this line of thought to be useful and if so we could try to implement some of these ideas.

thanks,

zach cp

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

[Matthias Bussonnier]

Le 2 déc. 2012 à 00:17, Zachary Charlop-Powers a écrit :

I recently posted a question to Stack Overflow about the possibility of using iPython to generate reproducible, repetitive reports. (http://stackoverflow.com/questions/13614783/programatically-add-cells-to-an-ipython-notebook-for-report-generation) Lets say that I have a number of Pandas data frames and for each data frame I want to generate a series of tables , graphs and text, and then combine these into a report. I could use a templating system like Jinja but I am curious to know if I could use the iPython framework of ipython-notebook ---> nbconverter ---> pdf/html/etc for the purpose of generating all of the data analyses and the report into a single document. I have followed Thomas K.'s suggestion to use the get_ipython().set_next_input() function to add text to the next code block. For my idea to work, a few things would need to work:

example code.

from IPython.core.display import Image, display_png

x = Image(filename='test.png')

for i in range(3):

x = Image(filename='/Users/zachpowers/Downloads/pics/Sieve_1.png') 

 get_ipython().set_next_input('Thats What I am Talking About!') 

 get_ipython().set_next_input(display_png(x))

if you run this you will see that there are a few different problems that you encounter.

1) you can't create Markdown cells  

We want to keep kernel separate from fronted, so we'll keep it as less aware as the frontend as possible. So adding MD cell to a kernel method is unlikely.

2) the display_png() function does not respect the .set_next_input() function so they are all displayed in the same output block in which they are called

Display_png does not 'return' a value to be displayed, it send the png to the display channel. 

So it is normal that it does not put the png in the next cell. 

Btw it is set next **input** so by definition is does only works on input so only with text. 

3) the .set_next_input() function would need to be modified to track which code blocks it insert.  It iterates over the loop and inserts a code block below the code block it sits in with the result that the code blocks are added in reverse order from expected.

same as 1.

4) there would need to be a way to hide the code-block that calls the rest of the functions.

5) you would probably still need a templating engine to generate interesting text of each markdown block

So in closing I think the python ---> nbconvert is a great tool and that it might potentially be useful for reproducible research of repetitive reports (say that five times fast). I think that a few tweaks to the ipython notebook might make this possible and that the benefit would be the creation of a single file from which analyses could be rerun and from which reports could be generated. However, upon writing up my thoughts I am not sure that this is easier than doing the analysis in one file and outputting the results via a separate templating system. 

IIUC, you are seeing this the wrong way. 

You can generate .ipynb file without ever going through the  browser.

Trying to "pilot" the browser to do that is overcomplicated.

ipynb files are "simple" Json files you can manipulate/create as you wish. 

What you want is a headless program that taking a template .ipynb files and somme command modify it. 

See some example here : 

http://wiki.ipython.org/Notebook_feedback

of gist modifying ipynb structure, or ever re-running ipynb files in a headless manner.

What do you think ?

-- 

Matthias

I hope some of the iPython community might find this line of thought to be useful and if so we could try to implement some of these ideas.

thanks,

zach cp

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user

[Zachary Charlop-Powers]

Matthias, Thanks for your reply. The re-send of my original e-mail was inadvertent. 

ipynb files are "simple" Json files you can manipulate/create as you wish. 

What you want is a headless program that taking a template .ipynb files and somme command modify it. 

See some example here : 

http://wiki.ipython.org/Notebook_feedback

of gist modifying ipynb structure, or ever re-running ipynb files in a headless manner.

What do you think ?

-- 

Matthias

I think that you are right that messing with the browser is more complicated than I need. I see that you can write code to generate or change the .ipynb Json format but if thats the case a simple Jinja template written to convert directly to HTML /RST/Latex will also suffice and will allow more control over the final appearance.

Thanks for your comment!

zach cp

_______________________________________________
IPython-User mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/ipython-user