Unable to see Sherlock terminal's modules in Jupyter notebook

[sakshamg94]

Hello! I am using Jupyter notebooks on Sherlock by following the steps detailed here (with your default container): https://vsoch.github.io/lessons/sherlock-jupyter/

But I am running into issues as I am unable to use the modules that I have loaded on the interactive terminal of my Sherlock login outside of this jupyter notebook. This screenshot in black is from my terminal, where I have listed the modules using module list. This other screenshot is from my jupyter notebook where I am trying to list modules the same way as in the terminal but getting a different list. Additionally, for some reason I am also unable to see the latest version of Python in the notebook using !python --version. Can you please help with what's going wrong?

[vsoch]

I don't work at Stanford anymore, but I'd be happy to help as best as I can, and I suggest you reach out to sherlock support after that. These lessons were written ~5 years ago and I left Stanford in February so things are likely to change!

But I am running into issues as I am unable to use the modules that I have loaded on the interactive terminal of my Sherlock login outside of this jupyter notebook.

So I suspect that forward is not sourcing your bash profile, where likely you have these things set up - it's sort of a raw/default you'd get without doing anything. Have you tried sourcing your profile? And I'm assuming that's where your custom modules are loaded?

!python --version

I'm pretty sure Kilian used to install python3, so try python3 --version instead.

[vsoch]

Also don't forget you are in a python session so you can just do:

import sys

sys.version_info
sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)

[sakshamg94]

Thanks for your prompt respose. Tried !python3 --version which returns Python 3.6.1 from the notebook. So that is good.

Tried

import sys
sys.version_info

and that returns sys.version_info(major=3, minor=6, micro=1, releaselevel='final', serial=0)

[vsoch]

Perfect! So that means the Python on the path is the one you are interacting with in the notebook, which is probably what you want :)

[sakshamg94]

Yes! But not sure why the modules are not syncing between what I see on my Sherlock terminal using module list and what I see in the notebook using module list (see screenshots from before). This is what is causing me all the problems as I am unable to use some python libraries

[vsoch]

Did you see my comment above? You probably aren't sourcing your bash profile, as you would when you login to an interactive session.

[sakshamg94]

When I do !source /home/users/sakshamg/.bashrc from Jupyter notebook, it returns /etc/profile.d/z99_srcc.sh: line 480: 'slurm-jobid_prpt': not a valid identifier

I checked that the .bashrc file is indeed located at /home/users/sakshamg/

[vsoch]

Then you probably want to look in that file and see exactly what modules are added (e.g., module use) and make sure to add that to your notebook.

And for production / your final code, I highly recommend you build (and run) a container to not rely on cluster-specific modules!

I can't comment beyond that - I'd recommend research computing support.

[sakshamg94]

Ok! I figured one way to solve this. The only modules that will be usable in the jupyter notebook will be the ones that go in the py3-jupyter.sbatch file which is used when submitting the job to the compute node. py3-jupyter.sbatch resides in the forward folder and needs to be modified to include other modules.