Running a Jupyter server
You can run a jupyter notebook in your browser from a compute node on the
cluster. This way your analysis runs on the file system where your data is, and
you can keep data, code and documentation in one place. slurm_jupyter
is a
script that starts and connects to a jupyter server on compute note and forwards
the web display to your local machine.
When you run slurm-jupyter
, it will connect to the cluster and write a
script that it submits on the cluster queue. Once that script runs on a compute
node, it starts the jupyter server for you. slurm-jupyter
then opens
connections so it can read the terminal output from jupyter and write it in the
teminal of your local macine as it normally happens when you run jupyter.
slurm-jupyter
. It then forwards a port form the cluster to your local
machine so you can see the jupyter web app in your local browser. The last thing
it does is to open the browser on your own machine and point it to the correct port.
Running slurm-jupyter
If your username on your local machine is the same as on the cluster, you can run slurm-jupyter like this:
slurm-jupyter -e monkey -A baboon
slurm-jupyter runs jupyterlab by default. If you want the classical notebook use this command:
slurm-jupyter -e monkey -A baboon --run notebook
The -e
option specifies some conda environment on the cluster that you
want jupyter to run in. Your notbooks run on the cluster, so that environment
needs should have jupyter/jupyterlab installed and any packages needed to run
your notebooks. Use the -A options to specify the project you want to bill
computing hours to. If your username on the cluster is different from that on
your local machine, you need to supply the that using the -u option. Use the
-h option to see help text and all the options.
Once executed `slurm-jupyter prints colored status messages to the terminal. All output from the jupyter server is redirected to your terminal too and printed without color.
After a while a jupyter notebook should show up in your browser window. The first time you do this, your browser may refuse to show jupyter because the connection is unsafe. In Safari you proceed to allow this. In Chrome, you can simply type the characters “thisisunsafe” while in the Chrome window:
Once ready, jupyter may ask for your password. To close the jupyter notebook, press Ctrl-c in the terminal. Closing the browser window does not close down the jupyter on the cluster.
The script slurm-jupyter
has a lot of options with sensible default values that you can see like this:
slurm-jupyter --help
Specifying resources
slurm-jupyter has a lot of options to specify required resources and the defaults are sensible. The most important ones to know are the ones that specify memory and time allotted for your session. For a jupyter session that can run up to five hours and needs eight gigabytes of memory:
slurm-jupyter -u hamlet -e monkey -A baboon -m 8g -t 5h