========== User Guide ========== .. contents:: Running pip =========== pip is a command line program. When you install pip, a ``pip`` command is added to your system, which can be run from the command prompt as follows: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip ``python -m pip`` executes pip using the Python interpreter you specified as python. So ``/usr/bin/python3.7 -m pip`` means you are executing pip for your interpreter located at /usr/bin/python3.7. .. group-tab:: Windows .. code-block:: shell py -m pip ``py -m pip`` executes pip using the latest Python interpreter you have installed. For more details, read the `Python Windows launcher`_ docs. Installing Packages =================== pip supports installing from `PyPI`_, version control, local projects, and directly from distribution files. The most common scenario is to install from `PyPI`_ using :ref:`Requirement Specifiers` .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install SomePackage # latest version python -m pip install SomePackage==1.0.4 # specific version python -m pip install 'SomePackage>=1.0.4' # minimum version .. group-tab:: Windows .. code-block:: shell py -m pip install SomePackage # latest version py -m pip install SomePackage==1.0.4 # specific version py -m pip install 'SomePackage>=1.0.4' # minimum version For more information and examples, see the :ref:`pip install` reference. .. _PyPI: https://pypi.org/ Basic Authentication Credentials ================================ pip supports basic authentication credentials. Basically, in the URL there is a username and password separated by ``:``. ``https://[username[:password]@]pypi.company.com/simple`` Certain special characters are not valid in the authentication part of URLs. If the user or password part of your login credentials contain any of the special characters `here `_ then they must be percent-encoded. For example, for a user with username "user" and password "he//o" accessing a repository at pypi.company.com, the index URL with credentials would look like: ``https://user:he%2F%2Fo@pypi.company.com`` Support for percent-encoded authentication in index URLs was added in pip 10.0.0 (in `#3236 `_). Users that must use authentication for their Python repository on systems with older pip versions should make the latest get-pip.py available in their environment to bootstrap pip to a recent-enough version. For indexes that only require single-part authentication tokens, provide the token as the "username" and do not provide a password, for example - ``https://0123456789abcdef@pypi.company.com`` netrc Support ------------- If no credentials are part of the URL, pip will attempt to get authentication credentials for the URL’s hostname from the user’s .netrc file. This behaviour comes from the underlying use of `requests`_ which in turn delegates it to the `Python standard library`_. The .netrc file contains login and initialization information used by the auto-login process. It resides in the user's home directory. The .netrc file format is simple. You specify lines with a machine name and follow that with lines for the login and password that are associated with that machine. Machine name is the hostname in your URL. An example .netrc for the host example.com with a user named 'daniel', using the password 'qwerty' would look like: .. code-block:: shell machine example.com login daniel password qwerty As mentioned in the `standard library docs `_, only ASCII characters are allowed. Whitespace and non-printable characters are not allowed in passwords. Keyring Support --------------- pip also supports credentials stored in your keyring using the `keyring`_ library. Note that ``keyring`` will need to be installed separately, as pip does not come with it included. .. code-block:: shell pip install keyring echo your-password | keyring set pypi.company.com your-username pip install your-package --extra-index-url https://pypi.company.com/ .. _keyring: https://pypi.org/project/keyring/ Using a Proxy Server ==================== When installing packages from `PyPI`_, pip requires internet access, which in many corporate environments requires an outbound HTTP proxy server. pip can be configured to connect through a proxy server in various ways: * using the ``--proxy`` command-line option to specify a proxy in the form ``[user:passwd@]proxy.server:port`` * using ``proxy`` in a :ref:`config-file` * by setting the standard environment-variables ``http_proxy``, ``https_proxy`` and ``no_proxy``. * using the environment variable ``PIP_USER_AGENT_USER_DATA`` to include a JSON-encoded string in the user-agent variable used in pip's requests. .. _`Requirements Files`: Requirements Files ================== "Requirements files" are files containing a list of items to be installed using :ref:`pip install` like so: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip install -r requirements.txt Details on the format of the files are here: :ref:`Requirements File Format`. Logically, a Requirements file is just a list of :ref:`pip install` arguments placed in a file. Note that you should not rely on the items in the file being installed by pip in any particular order. In practice, there are 4 common uses of Requirements files: 1. Requirements files are used to hold the result from :ref:`pip freeze` for the purpose of achieving :ref:`repeatable installations `. In this case, your requirement file contains a pinned version of everything that was installed when ``pip freeze`` was run. .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip freeze > requirements.txt python -m pip install -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip freeze > requirements.txt py -m pip install -r requirements.txt 2. Requirements files are used to force pip to properly resolve dependencies. As it is now, pip `doesn't have true dependency resolution `_, but instead simply uses the first specification it finds for a project. E.g. if ``pkg1`` requires ``pkg3>=1.0`` and ``pkg2`` requires ``pkg3>=1.0,<=2.0``, and if ``pkg1`` is resolved first, pip will only use ``pkg3>=1.0``, and could easily end up installing a version of ``pkg3`` that conflicts with the needs of ``pkg2``. To solve this problem, you can place ``pkg3>=1.0,<=2.0`` (i.e. the correct specification) into your requirements file directly along with the other top level requirements. Like so:: pkg1 pkg2 pkg3>=1.0,<=2.0 3. Requirements files are used to force pip to install an alternate version of a sub-dependency. For example, suppose ``ProjectA`` in your requirements file requires ``ProjectB``, but the latest version (v1.3) has a bug, you can force pip to accept earlier versions like so:: ProjectA ProjectB<1.3 4. Requirements files are used to override a dependency with a local patch that lives in version control. For example, suppose a dependency ``SomeDependency`` from PyPI has a bug, and you can't wait for an upstream fix. You could clone/copy the src, make the fix, and place it in VCS with the tag ``sometag``. You'd reference it in your requirements file with a line like so:: git+https://myvcs.com/some_dependency@sometag#egg=SomeDependency If ``SomeDependency`` was previously a top-level requirement in your requirements file, then **replace** that line with the new line. If ``SomeDependency`` is a sub-dependency, then **add** the new line. It's important to be clear that pip determines package dependencies using `install_requires metadata `_, not by discovering ``requirements.txt`` files embedded in projects. See also: * :ref:`Requirements File Format` * :ref:`pip freeze` * `"setup.py vs requirements.txt" (an article by Donald Stufft) `_ .. _`Constraints Files`: Constraints Files ================= Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. Their syntax and contents is nearly identical to :ref:`Requirements Files`. There is one key difference: Including a package in a constraints file does not trigger installation of the package. Use a constraints file like so: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install -c constraints.txt .. group-tab:: Windows .. code-block:: shell py -m pip install -c constraints.txt Constraints files are used for exactly the same reason as requirements files when you don't know exactly what things you want to install. For instance, say that the "helloworld" package doesn't work in your environment, so you have a local patched version. Some things you install depend on "helloworld", and some don't. One way to ensure that the patched version is used consistently is to manually audit the dependencies of everything you install, and if "helloworld" is present, write a requirements file to use when installing that thing. Constraints files offer a better way: write a single constraints file for your organisation and use that everywhere. If the thing being installed requires "helloworld" to be installed, your fixed version specified in your constraints file will be used. Constraints file support was added in pip 7.1. .. _`Installing from Wheels`: Installing from Wheels ====================== "Wheel" is a built, archive format that can greatly speed installation compared to building and installing from source archives. For more information, see the `Wheel docs `_ , :pep:`427`, and :pep:`425`. pip prefers Wheels where they are available. To disable this, use the :ref:`--no-binary ` flag for :ref:`pip install`. If no satisfactory wheels are found, pip will default to finding source archives. To install directly from a wheel archive: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install SomePackage-1.0-py2.py3-none-any.whl .. group-tab:: Windows .. code-block:: shell py -m pip install SomePackage-1.0-py2.py3-none-any.whl For the cases where wheels are not available, pip offers :ref:`pip wheel` as a convenience, to build wheels for all your requirements and dependencies. :ref:`pip wheel` requires the `wheel package `_ to be installed, which provides the "bdist_wheel" setuptools extension that it uses. To build wheels for your requirements and all their dependencies to a local directory: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install wheel python -m pip wheel --wheel-dir=/local/wheels -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip install wheel py -m pip wheel --wheel-dir=/local/wheels -r requirements.txt And *then* to install those requirements just using your local directory of wheels (and not from PyPI): .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install --no-index --find-links=/local/wheels -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip install --no-index --find-links=/local/wheels -r requirements.txt Uninstalling Packages ===================== pip is able to uninstall most packages like so: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip uninstall SomePackage .. group-tab:: Windows .. code-block:: shell py -m pip uninstall SomePackage pip also performs an automatic uninstall of an old version of a package before upgrading to a newer version. For more information and examples, see the :ref:`pip uninstall` reference. Listing Packages ================ To list installed packages: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip list docutils (0.9.1) Jinja2 (2.6) Pygments (1.5) Sphinx (1.1.2) .. group-tab:: Windows .. code-block:: console C:\> py -m pip list docutils (0.9.1) Jinja2 (2.6) Pygments (1.5) Sphinx (1.1.2) To list outdated packages, and show the latest version available: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip list --outdated docutils (Current: 0.9.1 Latest: 0.10) Sphinx (Current: 1.1.2 Latest: 1.1.3) .. group-tab:: Windows .. code-block:: console C:\> py -m pip list --outdated docutils (Current: 0.9.1 Latest: 0.10) Sphinx (Current: 1.1.2 Latest: 1.1.3) To show details about an installed package: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip show sphinx --- Name: Sphinx Version: 1.1.3 Location: /my/env/lib/pythonx.x/site-packages Requires: Pygments, Jinja2, docutils .. group-tab:: Windows .. code-block:: console C:\> py -m pip show sphinx --- Name: Sphinx Version: 1.1.3 Location: /my/env/lib/pythonx.x/site-packages Requires: Pygments, Jinja2, docutils For more information and examples, see the :ref:`pip list` and :ref:`pip show` reference pages. Searching for Packages ====================== pip can search `PyPI`_ for packages using the ``pip search`` command: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip search "query" .. group-tab:: Windows .. code-block:: shell py -m pip search "query" The query will be used to search the names and summaries of all packages. For more information and examples, see the :ref:`pip search` reference. .. _`Configuration`: Configuration ============= .. _config-file: Config file ----------- pip allows you to set all command line option defaults in a standard ini style config file. The names and locations of the configuration files vary slightly across platforms. You may have per-user, per-virtualenv or global (shared amongst all users) configuration: **Per-user**: * On Unix the default configuration file is: :file:`$HOME/.config/pip/pip.conf` which respects the ``XDG_CONFIG_HOME`` environment variable. * On macOS the configuration file is :file:`$HOME/Library/Application Support/pip/pip.conf` if directory ``$HOME/Library/Application Support/pip`` exists else :file:`$HOME/.config/pip/pip.conf`. * On Windows the configuration file is :file:`%APPDATA%\\pip\\pip.ini`. There are also a legacy per-user configuration file which is also respected, these are located at: * On Unix and macOS the configuration file is: :file:`$HOME/.pip/pip.conf` * On Windows the configuration file is: :file:`%HOME%\\pip\\pip.ini` You can set a custom path location for this config file using the environment variable ``PIP_CONFIG_FILE``. **Inside a virtualenv**: * On Unix and macOS the file is :file:`$VIRTUAL_ENV/pip.conf` * On Windows the file is: :file:`%VIRTUAL_ENV%\\pip.ini` **Global**: * On Unix the file may be located in :file:`/etc/pip.conf`. Alternatively it may be in a "pip" subdirectory of any of the paths set in the environment variable ``XDG_CONFIG_DIRS`` (if it exists), for example :file:`/etc/xdg/pip/pip.conf`. * On macOS the file is: :file:`/Library/Application Support/pip/pip.conf` * On Windows XP the file is: :file:`C:\\Documents and Settings\\All Users\\Application Data\\pip\\pip.ini` * On Windows 7 and later the file is hidden, but writeable at :file:`C:\\ProgramData\\pip\\pip.ini` * Global configuration is not supported on Windows Vista. The global configuration file is shared by all Python installations. If multiple configuration files are found by pip then they are combined in the following order: 1. The global file is read 2. The per-user file is read 3. The virtualenv-specific file is read Each file read overrides any values read from previous files, so if the global timeout is specified in both the global file and the per-user file then the latter value will be used. The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (``--index-url``) and set the HTTP timeout (``--default-timeout``) to 60 seconds your config file would look like this: .. code-block:: ini [global] timeout = 60 index-url = https://download.zope.org/ppix Each subcommand can be configured optionally in its own section so that every global setting with the same name will be overridden; e.g. decreasing the ``timeout`` to ``10`` seconds when running the ``freeze`` (:ref:`pip freeze`) command and using ``60`` seconds for all other commands is possible with: .. code-block:: ini [global] timeout = 60 [freeze] timeout = 10 Boolean options like ``--ignore-installed`` or ``--no-dependencies`` can be set like this: .. code-block:: ini [install] ignore-installed = true no-dependencies = yes To enable the boolean options ``--no-compile``, ``--no-warn-script-location`` and ``--no-cache-dir``, falsy values have to be used: .. code-block:: ini [global] no-cache-dir = false [install] no-compile = no no-warn-script-location = false For options which can be repeated like ``--verbose`` and ``--quiet``, a non-negative integer can be used to represent the level to be specified: .. code-block:: ini [global] quiet = 0 verbose = 2 It is possible to append values to a section within a configuration file such as the pip.ini file. This is applicable to appending options like ``--find-links`` or ``--trusted-host``, which can be written on multiple lines: .. code-block:: ini [global] find-links = http://download.example.com [install] find-links = http://mirror1.example.com http://mirror2.example.com trusted-host = http://mirror1.example.com http://mirror2.example.com This enables users to add additional values in the order of entry for such command line arguments. Environment Variables --------------------- pip's command line options can be set with environment variables using the format ``PIP_`` . Dashes (``-``) have to be replaced with underscores (``_``). For example, to set the default timeout: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell export PIP_DEFAULT_TIMEOUT=60 .. group-tab:: Windows .. code-block:: shell set PIP_DEFAULT_TIMEOUT=60 This is the same as passing the option to pip directly: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip --default-timeout=60 [...] .. group-tab:: Windows .. code-block:: shell py -m pip --default-timeout=60 [...] For command line options which can be repeated, use a space to separate multiple values. For example: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell export PIP_FIND_LINKS="http://mirror1.example.com http://mirror2.example.com" .. group-tab:: Windows .. code-block:: shell set PIP_FIND_LINKS="http://mirror1.example.com http://mirror2.example.com" is the same as calling: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install --find-links=http://mirror1.example.com --find-links=http://mirror2.example.com .. group-tab:: Windows .. code-block:: shell py -m pip install --find-links=http://mirror1.example.com --find-links=http://mirror2.example.com Options that do not take a value, but can be repeated (such as ``--verbose``) can be specified using the number of repetitions, so:: export PIP_VERBOSE=3 is the same as calling:: pip install -vvv .. note:: Environment variables set to be empty string will not be treated as false. Please use ``no``, ``false`` or ``0`` instead. .. _config-precedence: Config Precedence ----------------- Command line options have precedence over environment variables, which have precedence over the config file. Within the config file, command specific sections have precedence over the global section. Examples: - ``--host=foo`` overrides ``PIP_HOST=foo`` - ``PIP_HOST=foo`` overrides a config file with ``[global] host = foo`` - A command specific section in the config file ``[] host = bar`` overrides the option with same name in the ``[global]`` config file section Command Completion ================== pip comes with support for command line completion in bash, zsh and fish. To setup for bash:: python -m pip completion --bash >> ~/.profile To setup for zsh:: python -m pip completion --zsh >> ~/.zprofile To setup for fish:: python -m pip completion --fish > ~/.config/fish/completions/pip.fish Alternatively, you can use the result of the ``completion`` command directly with the eval function of your shell, e.g. by adding the following to your startup file:: eval "`pip completion --bash`" .. _`Installing from local packages`: Installing from local packages ============================== In some cases, you may want to install from local packages only, with no traffic to PyPI. First, download the archives that fulfill your requirements: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip download --destination-directory DIR -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip download --destination-directory DIR -r requirements.txt Note that ``pip download`` will look in your wheel cache first, before trying to download from PyPI. If you've never installed your requirements before, you won't have a wheel cache for those items. In that case, if some of your requirements don't come as wheels from PyPI, and you want wheels, then run this instead: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip wheel --wheel-dir DIR -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip wheel --wheel-dir DIR -r requirements.txt Then, to install from local only, you'll be using :ref:`--find-links ` and :ref:`--no-index ` like so: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install --no-index --find-links=DIR -r requirements.txt .. group-tab:: Windows .. code-block:: shell py -m pip install --no-index --find-links=DIR -r requirements.txt "Only if needed" Recursive Upgrade ================================== ``pip install --upgrade`` now has a ``--upgrade-strategy`` option which controls how pip handles upgrading of dependencies. There are 2 upgrade strategies supported: - ``eager``: upgrades all dependencies regardless of whether they still satisfy the new parent requirements - ``only-if-needed``: upgrades a dependency only if it does not satisfy the new parent requirements The default strategy is ``only-if-needed``. This was changed in pip 10.0 due to the breaking nature of ``eager`` when upgrading conflicting dependencies. As an historic note, an earlier "fix" for getting the ``only-if-needed`` behaviour was: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install --upgrade --no-deps SomePackage python -m pip install SomePackage .. group-tab:: Windows .. code-block:: shell py -m pip install --upgrade --no-deps SomePackage py -m pip install SomePackage A proposal for an ``upgrade-all`` command is being considered as a safer alternative to the behaviour of eager upgrading. User Installs ============= With Python 2.6 came the `"user scheme" for installation `_, which means that all Python distributions support an alternative install location that is specific to a user. The default location for each OS is explained in the python documentation for the `site.USER_BASE `_ variable. This mode of installation can be turned on by specifying the :ref:`--user ` option to ``pip install``. Moreover, the "user scheme" can be customized by setting the ``PYTHONUSERBASE`` environment variable, which updates the value of ``site.USER_BASE``. To install "SomePackage" into an environment with site.USER_BASE customized to '/myappenv', do the following: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell export PYTHONUSERBASE=/myappenv python -m pip install --user SomePackage .. group-tab:: Windows .. code-block:: shell set PYTHONUSERBASE=c:/myappenv py -m pip install --user SomePackage ``pip install --user`` follows four rules: #. When globally installed packages are on the python path, and they *conflict* with the installation requirements, they are ignored, and *not* uninstalled. #. When globally installed packages are on the python path, and they *satisfy* the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a ``--system-site-packages`` virtualenv). #. pip will not perform a ``--user`` install in a ``--no-site-packages`` virtualenv (i.e. the default kind of virtualenv), due to the user site not being on the python path. The installation would be pointless. #. In a ``--system-site-packages`` virtualenv, pip will not install a package that conflicts with a package in the virtualenv site-packages. The --user installation would lack sys.path precedence and be pointless. To make the rules clearer, here are some examples: From within a ``--no-site-packages`` virtualenv (i.e. the default kind): .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip install --user SomePackage Can not perform a '--user' install. User site-packages are not visible in this virtualenv. .. group-tab:: Windows .. code-block:: console C:\> py -m pip install --user SomePackage Can not perform a '--user' install. User site-packages are not visible in this virtualenv. From within a ``--system-site-packages`` virtualenv where ``SomePackage==0.3`` is already installed in the virtualenv: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip install --user SomePackage==0.4 Will not install to the user site because it will lack sys.path precedence .. group-tab:: Windows .. code-block:: console C:\> py -m pip install --user SomePackage==0.4 Will not install to the user site because it will lack sys.path precedence From within a real python, where ``SomePackage`` is *not* installed globally: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip install --user SomePackage [...] Successfully installed SomePackage .. group-tab:: Windows .. code-block:: console C:\> py -m pip install --user SomePackage [...] Successfully installed SomePackage From within a real python, where ``SomePackage`` *is* installed globally, but is *not* the latest version: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip install --user SomePackage [...] Requirement already satisfied (use --upgrade to upgrade) $ python -m pip install --user --upgrade SomePackage [...] Successfully installed SomePackage .. group-tab:: Windows .. code-block:: console C:\> py -m pip install --user SomePackage [...] Requirement already satisfied (use --upgrade to upgrade) C:\> py -m pip install --user --upgrade SomePackage [...] Successfully installed SomePackage From within a real python, where ``SomePackage`` *is* installed globally, and is the latest version: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: console $ python -m pip install --user SomePackage [...] Requirement already satisfied (use --upgrade to upgrade) $ python -m pip install --user --upgrade SomePackage [...] Requirement already up-to-date: SomePackage # force the install $ python -m pip install --user --ignore-installed SomePackage [...] Successfully installed SomePackage .. group-tab:: Windows .. code-block:: console C:\> py -m pip install --user SomePackage [...] Requirement already satisfied (use --upgrade to upgrade) C:\> py -m pip install --user --upgrade SomePackage [...] Requirement already up-to-date: SomePackage # force the install C:\> py -m pip install --user --ignore-installed SomePackage [...] Successfully installed SomePackage .. _`Repeatability`: Ensuring Repeatability ====================== pip can achieve various levels of repeatability: Pinned Version Numbers ---------------------- Pinning the versions of your dependencies in the requirements file protects you from bugs or incompatibilities in newly released versions:: SomePackage == 1.2.3 DependencyOfSomePackage == 4.5.6 Using :ref:`pip freeze` to generate the requirements file will ensure that not only the top-level dependencies are included but their sub-dependencies as well, and so on. Perform the installation using :ref:`--no-deps ` for an extra dose of insurance against installing anything not explicitly listed. This strategy is easy to implement and works across OSes and architectures. However, it trusts PyPI and the certificate authority chain. It also relies on indices and find-links locations not allowing packages to change without a version increase. (PyPI does protect against this.) Hash-checking Mode ------------------ Beyond pinning version numbers, you can add hashes against which to verify downloaded packages:: FooProject == 1.2 --hash=sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 This protects against a compromise of PyPI or the HTTPS certificate chain. It also guards against a package changing without its version number changing (on indexes that allow this). This approach is a good fit for automated server deployments. Hash-checking mode is a labor-saving alternative to running a private index server containing approved packages: it removes the need to upload packages, maintain ACLs, and keep an audit trail (which a VCS gives you on the requirements file for free). It can also substitute for a vendor library, providing easier upgrades and less VCS noise. It does not, of course, provide the availability benefits of a private index or a vendor library. For more, see :ref:`pip install\'s discussion of hash-checking mode `. .. _`Installation Bundle`: Installation Bundles -------------------- Using :ref:`pip wheel`, you can bundle up all of a project's dependencies, with any compilation done, into a single archive. This allows installation when index servers are unavailable and avoids time-consuming recompilation. Create an archive like this:: $ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX) $ python -m pip wheel -r requirements.txt --wheel-dir=$tempdir $ cwd=`pwd` $ (cd "$tempdir"; tar -cjvf "$cwd/bundled.tar.bz2" *) You can then install from the archive like this:: $ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX) $ (cd $tempdir; tar -xvf /path/to/bundled.tar.bz2) $ python -m pip install --force-reinstall --ignore-installed --upgrade --no-index --no-deps $tempdir/* Note that compiled packages are typically OS- and architecture-specific, so these archives are not necessarily portable across macOShines. Hash-checking mode can be used along with this method to ensure that future archives are built with identical packages. .. warning:: Finally, beware of the ``setup_requires`` keyword arg in :file:`setup.py`. The (rare) packages that use it will cause those dependencies to be downloaded by setuptools directly, skipping pip's protections. If you need to use such a package, see :ref:`Controlling setup_requires`. Fixing conflicting dependencies =============================== The purpose of this section of documentation is to provide practical suggestions to pip users who encounter an error where pip cannot install their specified packages due to conflicting dependencies (a ``ResolutionImpossible`` error). This documentation is specific to the new resolver, which you can use with the flag ``--use-feature=2020-resolver``. Understanding your error message -------------------------------- When you get a ``ResolutionImpossible`` error, you might see something like this: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install package_coffee==0.44.1 package_tea==4.3.0 .. group-tab:: Windows .. code-block:: shell py -m pip install package_coffee==0.44.1 package_tea==4.3.0 :: Due to conflicting dependencies pip cannot install package_coffee and package_tea: - package_coffee depends on package_water<3.0.0,>=2.4.2 - package_tea depends on package_water==2.3.1 In this example, pip cannot install the packages you have requested, because they each depend on different versions of the same package (``package_water``): - ``package_coffee`` version ``0.44.1`` depends on a version of ``package_water`` that is less than ``3.0.0`` but greater than or equal to ``2.4.2`` - ``package_tea`` version ``4.3.0`` depends on version ``2.3.1`` of ``package_water`` Sometimes these messages are straightforward to read, because they use commonly understood comparison operators to specify the required version (e.g. ``<`` or ``>``). However, Python packaging also supports some more complex ways for specifying package versions (e.g. ``~=`` or ``*``): +----------+---------------------------------+--------------------------------+ | Operator | Description | Example | +==========+=================================+================================+ | ``>`` | Any version greater than | ``>3.1``: any version | | | the specified version. | greater than ``3.1``. | +----------+---------------------------------+--------------------------------+ | ``<`` | Any version less than | ``<3.1``: any version | | | the specified version. | less than ``3.1``. | +----------+---------------------------------+--------------------------------+ | ``<=`` | Any version less than or | ``<=3.1``: any version | | | equal to the specified version. | less than or equal to ``3.1``. | +----------+---------------------------------+--------------------------------+ | ``>=`` | Any version greater than or | ``>=3.1``: | | | equal to the specified version. | version ``3.1`` and greater. | +----------+---------------------------------+--------------------------------+ | ``==`` | Exactly the specified version. | ``==3.1``: only ``3.1``. | +----------+---------------------------------+--------------------------------+ | ``!=`` | Any version not equal | ``!=3.1``: any version | | | to the specified version. | other than ``3.1``. | +----------+---------------------------------+--------------------------------+ | ``~=`` | Any compatible release. | ``~=3.1``: version ``3.1`` | | | Compatible releases are | or later, but not | | | releases that are within the | version ``4.0`` or later. | | | same major or minor version, | ``~=3.1.2``: version ``3.1.2`` | | | assuming the package author | or later, but not | | | is using semantic versioning. | version ``3.2.0`` or later. | +----------+---------------------------------+--------------------------------+ | ``*`` | Can be used at the end of | ``==3.1.*``: any version | | | a version number to represent | that starts with ``3.1``. | | | *all*, | Equivalent to ``~=3.1.0``. | +----------+---------------------------------+--------------------------------+ The detailed specification of supported comparison operators can be found in :pep:`440`. Possible solutions ------------------ The solution to your error will depend on your individual use case. Here are some things to try: 1. Audit your top level requirements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As a first step it is useful to audit your project and remove any unnecessary or out of date requirements (e.g. from your ``setup.py`` or ``requirements.txt`` files). Removing these can significantly reduce the complexity of your dependency tree, thereby reducing opportunities for conflicts to occur. 2. Loosen your top level requirements ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sometimes the packages that you have asked pip to install are incompatible because you have been too strict when you specified the package version. In our first example both ``package_coffee`` and ``package_tea`` have been *pinned* to use specific versions (``package_coffee==0.44.1b0 package_tea==4.3.0``). To find a version of both ``package_coffee`` and ``package_tea`` that depend on the same version of ``package_water``, you might consider: - Loosening the range of packages that you are prepared to install (e.g. ``pip install "package_coffee>0.44.*" "package_tea>4.0.0"``) - Asking pip to install *any* version of ``package_coffee`` and ``package_tea`` by removing the version specifiers altogether (e.g. ``python -m pip install package_coffee package_tea``) In the second case, pip will automatically find a version of both ``package_coffee`` and ``package_tea`` that depend on the same version of ``package_water``, installing: - ``package_coffee 0.46.0b0``, which depends on ``package_water 2.6.1`` - ``package_tea 4.3.0`` which *also* depends on ``package_water 2.6.1`` If you want to prioritize one package over another, you can add version specifiers to *only* the more important package: .. tabs:: .. group-tab:: Unix/macOS .. code-block:: shell python -m pip install package_coffee==0.44.1b0 package_tea .. group-tab:: Windows .. code-block:: shell py -m pip install package_coffee==0.44.1b0 package_tea This will result in: - ``package_coffee 0.44.1b0``, which depends on ``package_water 2.6.1`` - ``package_tea 4.1.3`` which also depends on ``package_water 2.6.1`` Now that you have resolved the issue, you can repin the compatible package versions as required. 3. Loosen the requirements of your dependencies ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Assuming that you cannot resolve the conflict by loosening the version of the package you require (as above), you can try to fix the issue on your *dependency* by: - Requesting that the package maintainers loosen *their* dependencies - Forking the package and loosening the dependencies yourself .. warning:: If you choose to fork the package yourself, you are *opting out* of any support provided by the package maintainers. Proceed at your own risk! 4. All requirements are loose, but a solution does not exist ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sometimes it's simply impossible to find a combination of package versions that do not conflict. Welcome to `dependency hell`_. In this situation, you could consider: - Using an alternative package, if that is acceptable for your project. See `Awesome Python`_ for similar packages. - Refactoring your project to reduce the number of dependencies (for example, by breaking up a monolithic code base into smaller pieces) Getting help ------------ If none of the suggestions above work for you, we recommend that you ask for help on: - `Python user Discourse`_ - `Python user forums`_ - `Python developers Slack channel`_ - `Python IRC`_ - `Stack Overflow`_ See `"How do I ask a good question?"`_ for tips on asking for help. Unfortunately, **the pip team cannot provide support for individual dependency conflict errors**. Please *only* open a ticket on the `pip issue tracker`_ if you believe that your problem has exposed a bug in pip. .. _dependency hell: https://en.wikipedia.org/wiki/Dependency_hell .. _Awesome Python: https://python.libhunt.com/ .. _Python user Discourse: https://discuss.python.org/c/users/7 .. _Python user forums: https://www.python.org/community/forums/ .. _Python developers Slack channel: https://pythondev.slack.com/ .. _Python IRC: https://www.python.org/community/irc/ .. _Stack Overflow: https://stackoverflow.com/questions/tagged/python .. _"How do I ask a good question?": https://stackoverflow.com/help/how-to-ask .. _pip issue tracker: https://github.com/pypa/pip/issues .. _`Using pip from your program`: Using pip from your program =========================== As noted previously, pip is a command line program. While it is implemented in Python, and so is available from your Python code via ``import pip``, you must not use pip's internal APIs in this way. There are a number of reasons for this: #. The pip code assumes that is in sole control of the global state of the program. pip manages things like the logging system configuration, or the values of the standard IO streams, without considering the possibility that user code might be affected. #. pip's code is *not* thread safe. If you were to run pip in a thread, there is no guarantee that either your code or pip's would work as you expect. #. pip assumes that once it has finished its work, the process will terminate. It doesn't need to handle the possibility that other code will continue to run after that point, so (for example) calling pip twice in the same process is likely to have issues. This does not mean that the pip developers are opposed in principle to the idea that pip could be used as a library - it's just that this isn't how it was written, and it would be a lot of work to redesign the internals for use as a library, handling all of the above issues, and designing a usable, robust and stable API that we could guarantee would remain available across multiple releases of pip. And we simply don't currently have the resources to even consider such a task. What this means in practice is that everything inside of pip is considered an implementation detail. Even the fact that the import name is ``pip`` is subject to change without notice. While we do try not to break things as much as possible, all the internal APIs can change at any time, for any reason. It also means that we generally *won't* fix issues that are a result of using pip in an unsupported way. It should also be noted that installing packages into ``sys.path`` in a running Python process is something that should only be done with care. The import system caches certain data, and installing new packages while a program is running may not always behave as expected. In practice, there is rarely an issue, but it is something to be aware of. Having said all of the above, it is worth covering the options available if you decide that you do want to run pip from within your program. The most reliable approach, and the one that is fully supported, is to run pip in a subprocess. This is easily done using the standard ``subprocess`` module:: subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package']) If you want to process the output further, use one of the other APIs in the module. We are using `freeze`_ here which outputs installed packages in requirements format.:: reqs = subprocess.check_output([sys.executable, '-m', 'pip', 'freeze']) If you don't want to use pip's command line functionality, but are rather trying to implement code that works with Python packages, their metadata, or PyPI, then you should consider other, supported, packages that offer this type of ability. Some examples that you could consider include: * ``packaging`` - Utilities to work with standard package metadata (versions, requirements, etc.) * ``setuptools`` (specifically ``pkg_resources``) - Functions for querying what packages the user has installed on their system. * ``distlib`` - Packaging and distribution utilities (including functions for interacting with PyPI). .. _`Resolver changes 2020`: Changes to the pip dependency resolver in 20.2 (2020) ===================================================== pip 20.1 included an alpha version of the new resolver (hidden behind an optional ``--unstable-feature=resolver`` flag). pip 20.2 removes that flag, and includes a robust beta of the new resolver (hidden behind an optional ``--use-feature=2020-resolver`` flag) that we encourage you to test. We will continue to improve the pip dependency resolver in response to testers' feedback. Please give us feedback through the `resolver testing survey`_. This will help us prepare to release pip 20.3, with the new resolver on by default, in October. Watch out for ------------- The big change in this release is to the pip dependency resolver within pip. Computers need to know the right order to install pieces of software ("to install ``x``, you need to install ``y`` first"). So, when Python programmers share software as packages, they have to precisely describe those installation prerequisites, and pip needs to navigate tricky situations where it's getting conflicting instructions. This new dependency resolver will make pip better at handling that tricky logic, and make pip easier for you to use and troubleshoot. The most significant changes to the resolver are: * It will **reduce inconsistency**: it will *no longer install a combination of packages that is mutually inconsistent*. In older versions of pip, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, in pip 20.0, ``pip install "six<1.12" "virtualenv==20.0.2"`` does the wrong thing, “successfully” installing ``six==1.11``, even though ``virtualenv==20.0.2`` requires ``six>=1.12.0,<2`` (`defined here `__). The new resolver, instead, outright rejects installing anything if it gets that input. * It will be **stricter** - if you ask pip to install two packages with incompatible requirements, it will refuse (rather than installing a broken combination, like it did in previous versions). So, if you have been using workarounds to force pip to deal with incompatible or inconsistent requirements combinations, now's a good time to fix the underlying problem in the packages, because pip will be stricter from here on out. This also means that, when you run a ``pip install`` command, pip only considers the packages you are installing in that command, and may break already-installed packages. It will not guarantee that your environment will be consistent all the time. If you ``pip install x`` and then ``pip install y``, it's possible that the version of ``y`` you get will be different than it would be if you had run ``pip install x y`` in a single command. We would like your thoughts on what pip's behavior should be; please answer `our survey on upgrades that create conflicts`_. We are also changing our support for :ref:`Constraints Files`: * Unnamed requirements are not allowed as constraints (see :issue:`6628` and :issue:`8210`) * Links are not allowed as constraints (see :issue:`8253`) * Constraints cannot have extras (see :issue:`6628`) How to test ----------- 1. **Install pip 20.2** with ``python -m pip install --upgrade pip``. 2. **Validate your current environment** by running ``pip check``. This will report if you have any inconsistencies in your set of installed packages. Having a clean installation will make it much less likely that you will hit issues when the new resolver is released (and may address hidden problems in your current environment!). If you run ``pip check`` and run into stuff you can’t figure out, please `ask for help in our issue tracker or chat `__. 3. **Test the new version of pip** (see below). To test the new resolver, use the ``--use-feature=2020-resolver`` flag, as in: ``pip install example --use-feature=2020-resolver`` The more feedback we can get, the more we can make sure that the final release is solid. (Only try the new resolver **in a non-production environment**, though - it isn't ready for you to rely on in production!) While we have tried to make sure that pip’s test suite covers as many cases as we can, we are very aware that there are people using pip with many different workflows and build processes, and we will not be able to cover all of those without your help. - If you use pip to install your software, try out the new resolver and let us know if it works for you with ``pip install``. Try: - installing several packages simultaneously - re-creating an environment using a ``requirements.txt`` file - using ``pip install --force-reinstall`` to check whether it does what you think it should - using constraints files - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need. - If you'd like pip to default to using the new resolver, run ``pip config set global.use-feature 2020-resolver`` (for more on that and the alternate ``PIP_USE_FEATURE`` environment variable option, see `issue 8661`_). - Run your project’s CI (test suite, build process, etc.) using the new resolver, and let us know of any issues. - If you have encountered resolver issues with pip in the past, check whether the new resolver fixes them. Also, let us know if the new resolver has issues with any workarounds you put in to address the current resolver’s limitations. We’ll need to ensure that people can transition off such workarounds smoothly. - If you develop or support a tool that wraps pip or uses it to deliver part of your functionality, please test your integration with pip 20.2. 4. **Please report bugs** through the `resolver testing survey`_. Setups we might need more testing on ------------------------------------ * Windows, including Windows Subsystem for Linux (WSL) * Macintosh * Debian, Fedora, Red Hat, CentOS, Mint, Arch, Raspbian, Gentoo * non-Latin localized filesystems and OSes, such as Japanese, Chinese, and Korean, and right-to-left such as Hebrew, Urdu, and Arabic * Multi-user installations * Requirements files with 100+ packages * An installation workflow that involves multiple requirements files * Requirements files that include hashes (:ref:`hash-checking mode`) or pinned dependencies (perhaps as output from ``pip-compile`` within ``pip-tools``) * Using :ref:`Constraints Files` * Continuous integration/continuous deployment setups * Installing from any kind of version control systems (i.e., Git, Subversion, Mercurial, or CVS), per :ref:`VCS Support` * Installing from source code held in local directories * Using the most recent versions of Python 3.6, 3.7, 3.8, and 3.9 * PyPy * Customized terminals (where you have modified how error messages and standard output display) Examples to try ^^^^^^^^^^^^^^^ Install: * `tensorflow`_ * ``hacking`` * ``pycodestyle`` * ``pandas`` * ``tablib`` * ``elasticsearch`` and ``requests`` together * ``six`` and ``cherrypy`` together * ``pip install flake8-import-order==0.17.1 flake8==3.5.0 --use-feature=2020-resolver`` * ``pip install tornado==5.0 sprockets.http==1.5.0 --use-feature=2020-resolver`` Try: * ``pip install`` * ``pip uninstall`` * ``pip check`` * ``pip cache`` Tell us about ------------- Specific things we'd love to get feedback on: * Cases where the new resolver produces the wrong result, obviously. We hope there won't be too many of these, but we'd like to trap such bugs now. * Cases where the resolver produced an error when you believe it should have been able to work out what to do. * Cases where the resolver gives an error because there's a problem with your requirements, but you need better information to work out what's wrong. * If you have workarounds to address issues with the current resolver, does the new resolver let you remove those workarounds? Tell us! Please let us know through the `resolver testing survey`_. Deprecation timeline -------------------- We plan for the resolver changeover to proceed as follows, using :ref:`Feature Flags` and following our :ref:`Release Cadence`: * pip 20.2: a beta of the new resolver is available, opt-in, using the flag ``--use-feature=2020-resolver``. pip defaults to legacy behavior. * pip 20.3: pip defaults to the new resolver, but a user can opt-out and choose the old resolver behavior, using the flag ``--use-deprecated=legacy-resolver``. * pip 21.0: pip uses new resolver, and the old resolver is no longer available. Since this work will not change user-visible behavior described in the pip documentation, this change is not covered by the :ref:`Deprecation Policy`. Context and followup -------------------- As discussed in `our announcement on the PSF blog`_, the pip team are in the process of developing a new "dependency resolver" (the part of pip that works out what to install based on your requirements). We're tracking our rollout in :issue:`6536` and you can watch for announcements on the `low-traffic packaging announcements list`_ and `the official Python blog`_. .. _freeze: https://pip.pypa.io/en/latest/reference/pip_freeze/ .. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en .. _issue 8661: https://github.com/pypa/pip/issues/8661 .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _tensorflow: https://pypi.org/project/tensorflow/ .. _low-traffic packaging announcements list: https://mail.python.org/mailman3/lists/pypi-announce.python.org/ .. _our survey on upgrades that create conflicts: https://docs.google.com/forms/d/e/1FAIpQLSeBkbhuIlSofXqCyhi3kGkLmtrpPOEBwr6iJA6SzHdxWKfqdA/viewform .. _the official Python blog: https://blog.python.org/ .. _requests: https://requests.readthedocs.io/en/master/user/authentication/#netrc-authentication .. _Python standard library: https://docs.python.org/3/library/netrc.html .. _Python Windows launcher: https://docs.python.org/3/using/windows.html#launcher