From 14506f82611c914d3190a301fa8d8b2d0ee49168 Mon Sep 17 00:00:00 2001 From: Erik Rose Date: Wed, 7 Oct 2015 16:35:27 -0400 Subject: [PATCH] Document hash-checking mode. --- docs/reference/pip_install.rst | 76 ++++++++++++++++++++++++++------ docs/user_guide.rst | 79 +++++++++++++++++++++++++--------- 2 files changed, 122 insertions(+), 33 deletions(-) diff --git a/docs/reference/pip_install.rst b/docs/reference/pip_install.rst index 3896d01b4..12626ea27 100644 --- a/docs/reference/pip_install.rst +++ b/docs/reference/pip_install.rst @@ -378,8 +378,8 @@ See the :ref:`pip install Examples`. SSL Certificate Verification ++++++++++++++++++++++++++++ -Starting with v1.3, pip provides SSL certificate verification over https, for the purpose -of providing secure, certified downloads from PyPI. +Starting with v1.3, pip provides SSL certificate verification over https, to +prevent man-in-the-middle attacks against PyPI downloads. .. _`Caching`: @@ -387,7 +387,7 @@ of providing secure, certified downloads from PyPI. Caching +++++++ -Starting with v6.0, pip provides an on by default cache which functions +Starting with v6.0, pip provides an on-by-default cache which functions similarly to that of a web browser. While the cache is on by default and is designed do the right thing by default you can disable the cache and always access PyPI by utilizing the ``--no-cache-dir`` option. @@ -443,19 +443,71 @@ When no wheels are found for an sdist, pip will attempt to build a wheel automatically and insert it into the wheel cache. -Hash Verification -+++++++++++++++++ +.. _`hash-checking mode`: -PyPI provides md5 hashes in the hash fragment of package download urls. +Hash-Checking Mode +++++++++++++++++++ -pip supports checking this, as well as any of the -guaranteed hashlib algorithms (sha1, sha224, sha384, sha256, sha512, md5). +Since version 8.0, pip can check downloaded package archives against local +hashes to protect against remote tampering. To verify a package against one or +more hashes, add them to the end of the line:: -The hash fragment is case sensitive (i.e. sha1 not SHA1). + FooProject == 1.2 --hash:sha256=2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 \ + --hash:sha256=486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7 -This check is only intended to provide basic download corruption protection. -It is not intended to provide security against tampering. For that, -see :ref:`SSL Certificate Verification` +(The ability to use multiple hashes is important when a package has both +binary and source distributions or when it offers binary distributions for a +variety of platforms.) + +The recommended hash algorithm at the moment is sha256, but stronger ones are +allowed, including all those supported by ``hashlib``. However, weak hashes +such as md5, sha1, and sha224 are excluded to avert false assurances of +security. + +Hash verification is an all-or-nothing proposition. Specifying a ``--hash`` +against any requirement not only checks that hash but also activates +*hash-checking mode*, which imposes several other security restrictions: + +* Hashes are required for all requirements. This is because a partially-hashed + requirements file is of little use and thus likely an error: a malicious + actor could slip bad code into the installation via one of the unhashed + requirements. Note that hashes embedded in URL-style requirements via the + ``#md5=...`` syntax suffice to satisfy this rule (regardless of hash + strength, for legacy reasons), though you use a stronger hash like sha256 + whenever possible. +* Hashes are required for all dependencies. An error is raised if there is a + dependency that is not spelled out and hashed in the requirements file. +* Requirements that take the form of project names (rather than URLs or local + filesystem paths) must be pinned to a specific version using ``==``. This + prevents a surprising hash mismatch upon the release of a new version + that matches the requirement specifier. +* ``--egg`` is disallowed, because it delegates installation of dependencies + to setuptools, giving up pip's ability to enforce any of the above. + +Hash-checking mode can be forced on with the ``--require-hashes`` command-line +option. This can be useful in deploy scripts, to ensure that the author of the +requirements file provided hashes. It is also a convenient way to bootstrap +your list of hashes, since it will show the hashes of the packages it +fetched. (It will fetch only a single archive for each package, so you may +still need to add additional hashes for alternatives: for instance if there is +both a binary and a source distribution available.) + +.. warning:: + Beware of the ``setup_requires`` keyword arg in :file:`setup.py`. The + (rare) packages that use it will cause those dependencies to be downloaded + by setuptools directly, skipping pip's hash-checking. If you need to use + such a package, see :ref:`Controlling + setup_requires`. + + +Hashes from PyPI +~~~~~~~~~~~~~~~~ + +PyPI provides an md5 hash in the fragment portion of each package download +URL. pip checks this as a protection against download corruption. However, +since the hash originates remotely, it is not a useful guard against tampering +and thus does not satisfy the ``--require-hashes`` demand that every package +have a local hash. .. _`editable-installs`: diff --git a/docs/user_guide.rst b/docs/user_guide.rst index 5711bd0e1..495a4a3d4 100644 --- a/docs/user_guide.rst +++ b/docs/user_guide.rst @@ -23,6 +23,8 @@ Specifiers` For more information and examples, see the :ref:`pip install` reference. +.. _PyPI: http://pypi.python.org/pypi + .. _`Requirements Files`: @@ -71,7 +73,6 @@ In practice, there are 4 common uses of Requirements files: pkg2 pkg3>=1.0,<=2.0 - 3. Requirements files are used to force pip to install an alternate version of a sub-dependency. For example, suppose `ProjectA` in your requirements file requires `ProjectB`, but the latest version (v1.3) has a bug, you can force @@ -591,44 +592,80 @@ From within a real python, where ``SomePackage`` *is* installed globally, and is Ensuring Repeatability ********************** -Four things are required to fully guarantee a repeatable installation using requirements files. +pip can achieve various levels of repeatability: -1. The requirements file was generated by ``pip freeze`` or you're sure it only - contains requirements that specify a specific version. +Pinned Version Numbers +---------------------- -2. The installation is performed using :ref:`--no-deps `. - This guarantees that only what is explicitly listed in the requirements file is - installed. +Pinning the versions of your dependencies in the requirements file +protects you from bugs or incompatibilities in newly released versions:: -3. None of the packages to be installed utilize the setup_requires keyword. See - :ref:`Controlling setup_requires`. + SomePackage == 1.2.3 + DependencyOfSomePackage == 4.5.6 -4. The installation is performed against an index or find-links location that is - guaranteed to *not* allow archives to be changed and updated without a - version increase. While this is safe on PyPI, it may not be safe for other - indices. If you are working with an unsafe index, consider the `peep project - `_ which offers this feature on top of pip - using requirements file comments. +Using :ref:`pip freeze` to generate the requirements file will ensure that not +only the top-level dependencies are included but their sub-dependencies as +well, and so on. Perform the installation using :ref:`--no-deps +` for an extra dose of insurance against installing +anything not explicitly listed. +This strategy is easy to implement and works across OSes and architectures. +However, it trusts PyPI, its CDN, and the network. It also relies on indices +and find-links locations not allowing packages to change without a version +increase. (PyPI does protect against this.) -.. _PyPI: http://pypi.python.org/pypi/ +Hash-checking Mode +------------------ +Beyond pinning version numbers, you can add hashes against which to verify +downloaded packages:: + + FooProject == 1.2 --hash:sha256=2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 + +This protects against compromises of PyPI, its CDN, the HTTPS certificate +chain, and the network between you and the packages. It also guards +against a package changing without a change in its version number, on +indexes that allow this. This approach is a good fit for automated +deployments to servers. + +Hash-checking mode is a labor-saving alternative to running an internal index +server containing approved packages: it removes the need to upload packages, +maintain ACLs, and keep an audit trail (which a VCS give you for the +requirements file for free). It can also substitute for a vendor library, +providing easier upgrades and less VCS noise. It does not, of course, +provide the availability benefits of an internal index or a vendor library. + +For more, see :ref:`pip install\'s discussion of hash-checking mode `. .. _`Installation Bundle`: -Create an Installation Bundle with Compiled Dependencies -******************************************************** +Installation Bundles +-------------------- -You can create a simple bundle that contains all of the dependencies you wish -to install using:: +An installation bundle is a single file containing all of a project's +dependencies, already compiled if applicable. They are useful for +installing when the index server is unavailable and for avoiding +recompilation. Create a bundle like this:: $ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX) $ pip wheel -r requirements.txt --wheel-dir=$tempdir $ cwd=`pwd` $ (cd "$tempdir"; tar -cjvf "$cwd/bundled.tar.bz2" *) -Once you have a bundle, you can then install it using:: +Once you have a bundle, you can then install it like this:: $ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX) $ (cd $tempdir; tar -xvf /path/to/bundled.tar.bz2) $ pip install --force-reinstall --ignore-installed --upgrade --no-index --no-deps $tempdir/* + +Compiled packages are typically OS- and architecture-specific, so bundles +containing them are as well. Hash-checking mode can be used with +installation bundles to ensure that future bundles are built with +identical packages. + +.. warning:: + Finally, beware of the ``setup_requires`` keyword arg in :file:`setup.py`. + The (rare) packages that use it will cause those dependencies to be + downloaded by setuptools directly, skipping pip's protections. If you need + to use such a package, see :ref:`Controlling + setup_requires`.