Jupyter notebooks on EMR

Explanatory data analysis requires interactive code execution. In case of spark and emr it is very convenient to run the code from jupyter notebooks on a remote cluster. EMR allows installing jupyter on the spark master. In order to do that configure "Applications" field for the emr cluster to contain also jupyter hub. For example: "Applications": [ { "Name": "Ganglia", "Version": "3.7.2" }, { "Name": "Spark", "Version": "2.4.0" }, { "Name": "Zeppelin", "Version": "0....

February 4, 2019 · SergeM

Interactive graphs in IPython

There are at least two libraries for ipython able to plot graphs interactively and inline. mpld3 bokeh I have tested only mpld3. It looks awesome. It implements exactly the thing I missed without matlab. Inside the notebook I now have a possibility to zoom and move plots. Of course, I could do it using qt mode, but it not so nice and convenient. Demo of mpld3: http://mpld3.github.io/_downloads/mpld3_demo.html...

June 7, 2015 · SergeM

Save Ipython notebook as script with the same filename

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 # creating a variable theNotebook with the name of notebook # source: http://stackoverflow.com/a/23619544 # In[1]: %%javascript var kernel = IPython.notebook.kernel; var thename = window.document.getElementById("notebook_name").innerHTML; var command = "theNotebook = " + "'"+thename+"'"; kernel.execute(command); # saving to a directory 'backup'. create the directory if it doesn't exist # source http://stackoverflow....

September 21, 2014 · SergeM

ipython. save notebook as script

Joint solution from [http://stackoverflow.com/a/23619544](http://stackoverflow.com/a/23619544" target="_blank) and [http://stackoverflow.com/a/19067979](http://stackoverflow.com/a/19067979" target="_blank) # [1] %%javascript var kernel = IPython.notebook.kernel; var thename = window.document.getElementById("notebook_name").innerHTML; var command = "theNotebook = " + "'"+thename+"'"; kernel.execute(command); [2] try : if(__IPYTHON__) : get_ipython().system(u’ipython nbconvert –to python {}.ipynb'.format(theNotebook)) except NameError : pass

August 31, 2014 · SergeM

displaying multiple grayscale figures in python's matplotlib

from matplotlib import pyplot as plt import matplotlib.cm as cm plt.figure() # without this it display one after another plt.imshow(image_one, cmap=cm.gray) # without cm.gray it displays grayscale images in colormap plt.imshow(image_two, cmap=cm.gray) # plt.show() use %matplotlib inline in ipython notebook to display image inplace Shorter version: %matplotlib inline import matplotlib.pyplot as plt plt.axis(‘off’) plt.imshow(dpt, cmap=plt.cm.gray, interpolation=‘nearest’)

June 6, 2014 · SM!