This is Data Science

Managing Virtual Environments with pyenv

A virtual environment is an isolated Python installation with its own set of installed packages. Creating one virtual environment per project ensures:

  • The packages installed for project A would never affect project B.
  • We can reproduce an exact dependency set from a requirements.txt file for any project.

Since the projects themselves are better managed when segregated across different directories, we would map the virtual environment also to the sub-directory created for every project. In chapter 9, we specifically describe the benefit of dividing the ~/Projects directory in two sub-directories — one for all organisational/work projects and one for all personal projects. In this chapter, we use a sub-directory called ~/Projects/personal/experiment_1 as a concrete example.

Creating a Virtual Environment

% cd Projects
% mkdir -p personal/experiment_1 && cd personal/experiment_1
% pyenv version
3.9.5 (set by /Users/username/Projects/.python-version)  # experiment_1 inherits the Projects-level default (3.9.5)
 
% pyenv local 3.8.10  # Override to 3.8.10 for this project
% pyenv virtualenv 3.8.10 venv-exp1  # Create the virtual environment
Requirement already satisfied: setuptools in .pyenv/versions/.../site-packages
Requirement already satisfied: pip in .pyenv/versions/.../site-packages
 
% pyenv local venv-exp1  # Activate it as the local default
(venv-exp1) username experiment_1 %

The (venv-exp1) prefix in the prompt confirms the environment is active. Automatic activation is provided by the eval "$(pyenv virtualenv-init -)" line that we inserted in ~/.zshrc file in chapter 5 — pyenv activates and deactivates environments as we enter and leave directories.

Installing Packages

Before running following command, note that we are already in ~/Projects/personal/experiment_1 directory, where virtual environment venv-exp1 is active.

% pip install numpy
Downloading numpy-2.0.2-cp39-cp39-macosx_14_0_arm64.whl (5.3 MB)
...
Successfully installed numpy-2.0.2

Here, numpy package is installed successfully inside the virtual environment venv-exp1 and is isolated to that environment. Attempting to import numpy outside venv-exp1 will raise a ModuleNotFoundError. In some cases (where Python was installed long time back or for some older versions), we may get an SSLError error message, for which troubleshooting is given in the next section, which can be skipped if the installation of numpy was successful.

Troubleshooting SSLError

At times, while trying to install a library we may see following error:

% pip install numpy
...
ERROR: Could not find a version that satisfies the requirement numpy (from versions: none)
ERROR: No matching distribution found for numpy
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

The SSLError error seen above means our Python installation is physically missing its connection to OpenSSL. This is a classic issue on macOS when installing Python versions via pyenv (3.8.10) without the proper build dependencies. It is further confirmed when we run the following command:

% python -c "import ssl; print(ssl.OPENSSL_VERSION)"
...
ImportError: dlopen(/Users/username/.pyenv/versions/3.8.10/lib/python3.8/lib-dynload/_ssl.cpython-39-darwin.so, 0x0002): Library not loaded: /opt/homebrew/opt/openssl@1.1/lib/libssl.1.1.dylib
...
(no such file, not in dyld cache)

To resolve the issue, we need to install macOS's SSL development libraries using Homebrew and force pyenv to rebuild our Python version while explicitly pointing to them.

% cd ~
% brew install openssl readline sqlite3 xz zlib
...
You have 10 outdated formulae installed.
openssl@3 3.4.0 is already installed but outdated (so it will be upgraded).
...

Once Homebrew has installed OpenSSL, we update following configurations:

% export PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"
% export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib"
% export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include"
% export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@3/lib/pkgconfig"

Now we can re-install the versions of Python that was having the library installation issue:

pyenv install --force 3.8.10

Once Python is re-installed, we need to do same with virtual environment — here we first delete existing environment and then re-create it.

% cd Projects/personal/experiment_1
% rm -rf /Users/username/.pyenv/versions/3.8.10/envs/venv-exp1
% rm -rf /Users/username/.pyenv/versions/venv-exp1  
% pyenv virtualenv 3.8.10 venv-exp1
Requirement already satisfied: setuptools in .pyenv/versions/.../site-packages
Requirement already satisfied: pip in .pyenv/versions/.../site-packages

This solves the SSLError issue. We should also make sure that pypi.org is added as our trusted host in ~/.config/pip/pip.conf file:

[global]
trusted-host: pypi.python.org pypi.org files.pythonhosted.org

Finally, let's try to install the library again. Note that this must be run from ~/Projects/personal/experiment_1 directory to ensure installation is within venv-exp1 environment.

% pip install numpy                                            
Collecting numpy
  Downloading numpy-2.0.2-cp39-cp39-macosx_14_0_arm64.whl (5.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 35.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.2

Running Python Scripts Inside the Environment

Let's start with creating a Python script file ~/Projects/personal/experiment_1/script1.py with a basic content:

try:
  import numpy as np
  a = np.array([1, 2, 3]) * 2
  print("success")
except:
  print("failure")

Once the script is ready we can run it from ~/Projects/personal/experiment_1 directory:

% python script1.py
success

The "success" message above shows that the numpy library was imported and used successfully.

We can also start an interactive console pointing to the virtual environment as shown below:

% pyenv which python
/Users/username/.pyenv/versions/venv-exp1/bin/python
 
% ~/.pyenv/versions/venv-exp1/bin/python
Python 3.8.10
>>> import numpy as np
>>> np.array([1, 2, 3]) * 2
array([2, 4, 6])
>>> exit()

This is all about managing virtual environment with pyenv at project level. In next chapter, we discuss managing notebboks associated with different kernels, which in turn, are mapped to different virtual environements.