Managing Kernels and Notebooks with pyenv

Jupyter notebooks execute code through a kernel — a background process that runs Python and returns results to the browser. Each kernel maps to a specific Python installation. By creating a dedicated kernel per virtual environment, we can switch between completely isolated Python configurations within the same JupyterLab session.

Installing JupyterLab

We should install JupyterLab under the global environment (which is mapped to Python 3.8.10):

% cd ~
% pip install jupyterlab
Collecting jupyterlab
  Downloading jupyterlab-3.0.16-py3-none-any.whl (8.2 MB)
     |████████████████████████████████| 8.2 MB 1.1 MB/s
...
Successfully installed MarkupSafe-2.0.1 ... jupyter-client-6.1.12 jupyter-core-4.7.1 jupyter-server-1.8.0 jupyterlab-3.0.16 jupyterlab-pygments-0.1.2 jupyterlab-server-2.6.0
...
 
% pip list | grep jupyter
jupyterlab          3.0.16
jupyterlab-server   2.6.0

Creating a Project Virtual Environment

In the previous chapter, we already created a virtual environemnet, which we can be reused. However, here we create another project directory and associate another virtual environment:

% cd Projects/personal
% mkdir experiment_2 && cd experiment_2
 
% pyenv virtualenv 3.9.5 venv-exp2
% pyenv local venv-exp2
(venv-exp2) username experiment_2 %

Installing and Registering a Jupyter Kernel

Before running following command, note that we are already in ../Projects/personal/experiment_2 directory, where virtual environment venv-exp2 is already active.

% pip install ipykernel
Collecting ipykernel
  Using cached ipykernel-5.5.5-py3-none-any.whl (120 kB)
Installing collected packages: ipython-genutils, ..., jupyter-client, ipython, ipykernel
Successfully installed appnope-0.1.2 backcall-0.2.0 decorator-5.0.9 ipykernel-5.5.5 ...

While installing kernels, it makes changes in /Users/username/.pyenv/versions/3.9.5/lib/python3.9/site-packages, indicating that kernel needs to be separately installed for each version of python managed by pyenv. Since we are currently in ../Projects/personal/experiment_2 directory (defaulting to venv-exp2 which is further mapped to python-3.9.5), kernel is installed for this version only. Now let's register the installed kernel.

% python -m ipykernel install --user \
    --name exp2 --display-name "Python3.9.5 (exp2)"
Installed kernelspec exp2 in /Users/username/Library/Jupyter/kernels/exp2

The --name flag sets the internal kernel identifier; --display-name is what appears in the JupyterLab kernel picker. We should use descriptive names so that it is immediately clear which environment each kernel maps to.

Launching JupyterLab

JupyterLab must be installed in the environment from which we launch it. If we directly try to launch jupyterlab from current environment, we see following error:

% jupyter lab
pyenv: jupyter-lab: command not found

The `jupyter-lab' command exists in these Python versions:
  3.8.10

This is because our JupyterLab is installed globally. We can switch to the home directory and launch jupyterlab from there, and still be able to select venv-exp2 environment for a specific notebook:

% cd ~
% jupyter lab

Once JupyterLab opens, all registered kernels appear in the launcher. We can select Python3.9.5 (exp2) to work inside the isolated environment. Packages installed in venv-exp2 will be available only in that kernel, confirming proper isolation.

Note: A common mistake is installing ipykernel before creating the virtual environment. If we do this, the kernel points to the base Python version rather than the isolated environment, and packages bleed across sub-directories that share the same Python version. We must always create the virtual environment first.