venv-update: quick, exact

Issues | Github | CI | PyPI

Release v3.1.1 (Installation)

Introduction

venv-update is an MIT-Licensed tool to quickly and exactly synchronize a large python project’s virtualenv with its requirements.

This project ships as two separable components: pip-faster and venv-update.

Both are designed for use on large projects with hundreds of requirements and are used daily by Yelp engineers.

Why?

Generating a repeatable build of a virtualenv has many edge cases. If a requirement is removed, it should be uninstalled when the virtualenv is updated. If the version of python has changed, the only reliable solution is to re-build the virtualenv from scratch. Initially, this was exactly how we implemented updates of our virtualenv, but it slowed things down terribly. venv-update handles all of these edge cases and more, without completely starting from scratch (in the usual case).

In a large application, best practice is to “pin” versions, with requirements like package-x==1.2.3 in order to ensure that dev, staging, test, and production will all use the same code. Currently pip will always reach out to PyPI to list the versions of package-x regardless of whether the package is already installed, or whether its wheel can be found in the local cache. pip-faster adds these optimizations and others.

venv-update

A small script designed to keep a virtualenv in sync with a changing list of requirements. The contract of venv-update is this:

The virtualenv state will be exactly the same as if you deleted and rebuilt it from scratch, but will get there in much less time.

The needs of venv-update are what drove the development of pip-faster. For more, see venv-update in detail.

pip-faster

pip-faster is a drop-in replacement for pip. pip-faster’s contract is:

Take the same arguments and give the same results as pip, just more quickly.

This is especially true in the case of pinned requirements (e.g. package-x==1.2.3). If you’re also using venv-update (which we heartily recommend!), you can view pip-faster as an implementation detail. For more, see pip-faster in detail.

How much faster?

If we install plone (a large python application with more than 250 dependencies) we get these numbers:

testcase	pip v8.0.2	pip-faster	improvement
cold	4m 39s	4m 16s	8%
noop	7.11s	2.40s	196%
warm	44.6s	21.3s	109%

In the “cold” case, all caches are completely empty. In the “noop” case nothing needs to be done in order to update the virtualenv. In the “warm” case caches are fully populated, but the virtualenv has been completely deleted.

The Benchmarks page has more detail.

Installation

Because venv-update is meant to be the entry-point for creating your virtualenv directory and installing your packages, it’s not meant to be installed via pip; that would require a virtualenv to already exist!

Instead, the script is designed to be vendored (directly checked in) to your project. It has no dependencies other than virtualenv and the standard Python library.

curl -o venv-update https://raw.githubusercontent.com/Yelp/venv-update/v3.1.1/venv_update.py
chmod +x venv-update

Usage

By default, running venv-update will create a virtualenv named venv in the current directory, using requirements.txt in the current directory. This should be the desired default for most projects.

If you need more control, you can pass additional options to both virtualenv and pip. The command-line help gives more detail:

$ venv-update --help
usage: venv-update [-hV] [options]

Update a (possibly non-existent) virtualenv directory using a pip requirements
file.  When this script completes, the virtualenv directory should contain the
same packages as if it were deleted then rebuilt.

venv-update uses "trailing equal" options (e.g. venv=) to delimit groups of
(conventional, dashed) options to pass to wrapped commands (virtualenv and pip).

Options:
    venv=             parameters are passed to virtualenv
                       default: venv
    install=          options to pip-command
                       default: -r requirements.txt
    pip-command=      is run after the virtualenv directory is bootstrapped
                       default: pip-faster install --upgrade --prune
    bootstrap-deps=   dependencies to install before pip-command= is run
                       default: venv-update==3.1.1

Examples:
    # install requirements.txt to "venv"
    venv-update

    # install requirements.txt to "myenv"
    venv-update venv= myenv

    # install requirements.txt to "myenv" using Python 3.4
    venv-update venv= -ppython3.4 myenv

    # install myreqs.txt to "venv"
    venv-update install= -r myreqs.txt

    # install requirements.txt to "venv", verbosely
    venv-update venv= venv -vvv install= -r requirements.txt -vvv

    # install requirements.txt to "venv", without pip-faster --update --prune
    venv-update pip-command= pip install

We strongly recommend that you keep the default value of pip-command= in order
to quickly and reproducibly install your requirements. You can override the
packages installed during bootstrapping, prior to pip-command=, by setting
bootstrap-deps=

Pip options are also controllable via environment variables.
See https://pip.readthedocs.org/en/stable/user_guide/#environment-variables
For example:
    PIP_INDEX_URL=https://pypi.example.com/simple venv-update

Please send issues to: https://github.com/yelp/venv-update

… in your Makefile

venv-update is a good fit for use with make because it is idempotent and should never fail, under normal circumstances. Here’s an example Makefile:

venv: requirements.txt
   ./bin/venv-update

.PHONY: run-some-script
run-some-script: venv
   ./venv/bin/some-script

… with tox

tox is a useful tool for testing libraries against multiple versions of the Python interpreter. You can speed it up by telling it to use venv-update for dependency installation; not only will it avoid network access and prefer wheels, but it’s also better at syncing a virtualenv (whereas tox will often throw out an entire virtualenv and start over).

To start using venv-update inside tox, copy the venv-update script into your project (for example, at bin/venv-update).

Then, apply a change like this to your tox.ini file:

  [tox]
  envlist = py27,py34
+ skipsdist = true

  [testenv]
+ venv_update =
+     {toxinidir}/bin/venv-update \
+        venv= {envdir} \
+        install= -r {toxinidir}/requirements.txt {toxinidir}
- deps = -rrequirements.txt
  commands =
+     {[testenv]venv_update}
      py.test tests/
      pre-commit run --all-files

The exact changes will of course vary, but above is a general template. The two changes are: running venv-update as the first test command, and removing the list of deps (so that tox will never invalidate your virtualenv itself; we want to let venv-update manage that instead). The skipsdist avoids installing your package twice. In tox<2, it also prevents all of your packages dependencies from being installed by pip-slower.