Three questions on using EGI's JupyterLab

Hi and thank you very much for creating the EGI Notebooks service!

Please allow me to expose few problems I’m having with the service:

My first question is about the problem I’m having with displaying interactive plots using %matplotlib widget, I get the following message instead of the plot: “Error creating widget: could not find model”. On the other hand, no problems with displaying simple widgets like sliders.

My second question is about adding new modules (something I have to do in order to make my notebooks work). What I did yesterday is that I installed a bunch of new packages, updated most of the existing ones etc… (using the File->New->Terminal). The problem is that today - it all came back to the original state - none of the modifications I did were left. Is this the intended behavior? If so, is there a way of creating the persistent working environment? I also tried uploading my environment and then using conda env create -f environment.yml , but different errors occurred, so no luck there neither.

Third question is about sharing the notebooks. Does every person I want to share the notebook with need to have EGI account? I tried accessing my notebook using my google account with EGI, but wasn’t able to.

Thanks in advance, Cheers, Dejan.

Hi @shcrela! Thanks very much for this feedback - it will really help the service developers and other users in the long run. We’ve discussed this a bit and a full response – as well as hopefully some improvements to the service based on your suggestions – will take some time. In the meantime, I wanted to keep the discussion going.

First of all, could you give us some context on how you arrived at these issues? Were you following a tutorial somewhere? I would like to be able to reproduce this error you describe:

Secondly,

I think this is indeed the expected behaviour. See

and

The persistent storage is available -

but I’m not sure if this is true for all deployments. In my workspace, I had included extra modules (dask, toil and others) - and they are still in the environment, so I’m not able to reproduce your experience – but again, this can be explained if I’m using a different virtual organisation than you are.

This is interesting - we might want to split that off in another topic.

Ah! I had the same question in the tutorial - @enolfc response was,

This is definitely a feature that the community would benefit from, so if you want it perhaps you could help us drive the demand. This could be done by starting a topic in the notebooks category describing it and we could get folks to vote it up.

1 Like

Hi @brucellino, and thank you for your reply.

To keep the discussion going, as you suggested, here are some additional details on the problems I evoked:

First of all, could you give us some context on how you arrived at these issues? Were you following a tutorial somewhere? I would like to be able to reproduce this error you describe:

Well, I had to do a presentation this Monday on some work I did on segmenting the grains observed in the AFM images of certain thin films, and I thought of using the EGI Notebooks service (giving you guys some well deserved free advertising at the same time :wink:). So the idea was to just upload the notebook I had on my PC, already prepared, and clone my environment (or install manually the modules I needed) and the script would work :unicorn:.

Ideally, I wanted also to give my colleagues the possibility to run the script themselves online, allowing them to upload their own images to be processed :trophy: (That’s where my third question about sharing the notebooks came from).

So here is a simple example to reproduce the error of interactive plots not displaying:

 %matplotlib widget
 from ipywidgets import widgets, interact
 import matplotlib.pyplot as plt
 from skimage import morphology

And then:

test_img = morphology.disk(11)
plt.figure()
plt.imshow(test_img)
plt.show()

It shows: Error displaying widget On the other hand:

def f(x):
    return x
interact(f, x=10);

Displays the slider, as it was supposed to.

N.B. For the %matplotlib widget magic to work, one needs to have node.js installed, as well as ipympl, both of which I did on the terminal with the following:

sudo apt-get update
sudo apt-get install nodejs
conda install -c conda-forge ipympl

I don’t know if you already had questions about displaying interactive plots, but personally, I consider them as “a must have” option, so any effort on making them available would be greatly appreciated.

Further information:

  • Commenting out the %matplotlib widget or switching to inline displays the plots inline with no problem, but alas, with no interactivity
  • Changing to %matplotlib notebook or nbagg gives the message: JavaScript output is disabled in JupyterLab

As for the question about persistently changing the environment (adding new modules etc.) I’m not sure if I understand you correctly. The folder “persistent” does seem to retain all the files it contains, but it seems to me that all the tweaks in the configuration of the virtual environment (I’m using here the terms I don’t fully comprehend) are lost after a certain time of inactivity (the day after, for example).

Thanks again for all the work you’ve done so far!

Cheers, Dejan.

1 Like

Hi @shcrela! and thanks again for the feedback

The service is designed to kill the notebooks after some inactivity period. Killing the notebooks implies that any changes to the container (e.g. installing package) will be lost, although the persistent folder will be kept. You could install python modules into that folder, but if you require anything that modifies the system (e.g. apt-get install) then that won’t persist.

I’m still not sure on what’s the best solution to the problem, one possibility that we can support easily is to provide us with a docker container with the right environment so you will have it customised to your exact needs. Issue with that solution is that is not that easy to make a nice UI to handle that (specially when there are errors)

Another option is to submit a PR to include more stuff into the base image:

So every user would have these included (if it’s general enough, I’m happy to take them)

The third possibility is to use something like mybinder.org, but there are some pieces still missing to make it work with authentication

I will check the conda env create -f environment.yml command and see if I can fix the errors.

1 Like

Hi @shcrela!

Can you try to run your notebook again? I have included the missing packages to the base image. This should at least allow you to get started.

Could you also provide a sample environment.yml for the conda command?

Thanks!

1 Like

Hi @enolfc,

Thanks a lot for having looked into this.

The first message I get when connecting to the Notebook, is that matplotlib widgets needs to be included (or something like that); The automatic build failed, so I need to run manually jupyter lab build command on the terminal. However, it too fails to build. Since unfortunately, copy/paste doesn’t seem to work in the terminal, I can’t show you all the output form the terminal, but I could perhaps send you the yarn.log file, if it can help. The build stalls and then gets killed at $ webpack --config webpack.prod.config.js Perhaps updating yarn could solve the problem?

As for the environment.yml I’ll try to clean it up first, then try it on mybinder.org just to be sure to send you the clean working version.

Thanks,

Best regards, Dejan.

Hi again @enolfc,

So I just did some testing on mybinder.org and it worked quite well. So here are the contents of my environment.yml file:

name: Segmentation
channels:
  - conda-forge
dependencies:
  - bqplot
  - scikit-learn
  - scipy
  - numpy
  - matplotlib
  - scikit-image
  - ipywidgets
  - ipympl
  - pip:
      - opencv-python

Here’s also the link to the project: https://github.com/shcrela/Segmentation , if it interests you

(I can’t make much use of mybinder though, because I wouldn’t know how to upload the files to be treated; EGI Notebooks service is in that regard much superior with its persistent storage.) If you’d allow me to develop the reasoning, for this particular use-case, the main disadvantage of the service would be somewhat complicated access to the notebook for the people only wishing to have a peek at the script at work.

Thanks a lot,

Dejan.

Hi Dejan,

I have been dragged by other stuff these days and was not able to further look into the issue. I will try to do some more testing by the end of the week. Will keep you posted!

Thanks Enol