Managing virtual environments in python

Table of contents

Writing software in python is a quick and easy process: New projects don't require special setup and missing packages can be installed with pip. But when maintaining multiple projects on the same machine, you quickly run into issues: some might require conflicting versions of the same library or different python versions. Virtual environments are used to solve the issues.

What is a virtual environment?

At it's core, a virtual python environment is just a directory that isolates a python project form the rest of the machine it is stored on: the directory contains it's own python interpreter and installed packages, so the project does not need to rely on globally installed versions. This helps developers maintain many working environments for many different projects, without compromising on the host installation or projects affecting each other.

Creating a virtual environment

The default tool of choice for virtual environments is venv, mainly because it adheres to the current standards and is included out of the box in python 3.3 and later versions.

To create a virtual environment, you first need to choose a name for it's directory (where interpreter and packages will be stored). It is common to also name this directory venv, to make it easy for other users to know immediately that this is the virtual environment directory, managed by venv, just by reading the directory name.

Creating a new virtual environment is as simple as:

python -m venv venv

This will create a directory named venv in the current directory, containing all packages, interpreters and executable scripts needed to isolate it from the host system.

Activating the virtual environment

Just creating the virtual environment doesn't change the way we interact with files inside it. To get the benefits of using venv, we must now activate the virtual environment. This is done by using source from a bash shell:

source venv/bin/activate

Or running the batch file on windows:

venv\Scripts\activate.bat

Be careful to use source when activating the environment in bash: you cannot execute the bash files activate directly, you must source it to get it working properly!

You need to activate the virtual environment every time you open a new terminal. To deactivate it, you can either close the terminal, or run:

deactivate

Managing dependencies with venv

Once the virtual environment is activated, using venv is seamless: it quietly adjusts pip and python commands to use the virtual environment versions without any additional setup. New users may be confused initially, because the virtual environment does not contain any packages initially (and doesn't inherit system packages either), so the first steps are usually to install some packages, even if they were already present on the host system.

Installing a package works as usual:

pip install requests

The command looks and behaves the same, but packages are now installed into venv/lib and venv/lib64, respectively.

Fixing dependency versions

When installing packages into a virtual environment, package versions won't change without manual updates anymore. While this works as intended on your local machine, this may not be the case when sharing the code, for example by uploading it to a public git repository. You could upload the entire venv directory, but then your project can only be run by users that have the same operating system and processor architecture as you, for example mac users can't user windows projects and vice versa.

That's not intuitive, so the convention became to not include the venv/ directory in the source code, and instead provide a text file named requirements.txt that contains the names and versions of all packages required to run the project. This file can be quickly generated from your virtual environment:

pip freeze > requirements.txt

The only thing you need to do now is re-run this command when you install or update packages.

Users that want to install the project on another machine can first create and activate a new virtual environment, and then install all packages at once:

pip install -r requirements.txt

Removing packages

Especially for longer-running projects, it is vital to be precise with package management. Every package included in the virtual environment will be installed on all user's machines if they run the project - whether your code actually uses it or not. To reduce bloat, you should remove packages when you don't need them anymore:

pip uninstall requests

This will delete the files from your local venv/ directory, and running pip freeze will not list it as a requirement for installation anymore.



Managing dependencies in virtual environments with venv can be a quick and easy solution, but it requires diligence on the part of the developer: While installing dependencies is easy, forgetting to uninstall obsolete ones can quickly bloat the project and figuring out which packages are or aren't required is tedious to do in hindsight. Some package managers seek to remedy this problem, the most popular ones being poetry and pdm. While these tools provide further automation for dependency management, they are not widely adopted throughout the open source community at this point, so understanding venv remains a vital and valuable skill to have.

More articles

Generator functions in python

Generating sequences on the fly, one by one

Understanding  Go Closures

When two funcs aren't quite the same

Designing REST APIs for long-running tasks

Going beyond the request-response pattern

Working with tar archives

...and tar.gz, tar.bz2 and tar.xz

Understanding the linux tee command

Copying stdin to multiple outputs

Protecting linux servers from malware with ClamAV and rkhunter

Finding malicious files the open source way