From f74c764c851fbce58aa4b5ed8df0f49ae8671790 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Wed, 24 Jun 2020 00:20:14 -0400 Subject: [PATCH 01/18] Work in progress: docs: 20.2 beta testing guide --- docs/html/user_guide.rst | 105 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index f2e49fadf..96e3e9350 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1023,4 +1023,109 @@ of ability. Some examples that you could consider include: * ``distlib`` - Packaging and distribution utilities (including functions for interacting with PyPI). +Testing the 20.2 beta +===================== + +Please test pip 20.2b2. + +Watch out for +------------- + +The big thing in this beta is changes to the pip dependency resolver within pip. + +Computers need to know the right order to install pieces of software +("to install `x`, you need to install `y` first"). So, when Python +programmers share software as packages, they have to precisely describe +those installation prerequisites, and pip needs to navigate tricky +situations where it's getting conflicting instructions. This new +dependency resolver will make pip better at handling that tricky +logic, and easier for you to use and troubleshoot. + +The most significant changes to the resolver are: + +* It will **reduce inconsistency**: it will *no longer install a +combination of packages that is mutually inconsistent*. In older +versions of pip, it is possible for pip to install a package which +does not satisfy the declared requirements of another installed +package. For example, in pip 20.0, ``pip install "six<1.12" +"virtualenv==20.0.2"`` does the wrong thing, “successfully” installing +``six==1.11``, even though ``virtualenv==20.0.2`` requires +``six>=1.12.0,<2`` (`defined here +`__). +The new resolver, instead, outright rejects installing anything +if it gets that input. + +* It will be **stricter** - if you ask pip to install two packages with +incompatible requirements, it will refuse (rather than installing a +broken combination, like it did in previous versions). + +So, if you have been using workarounds to force pip to deal with +incompatible or inconsistent requirements combinations, now's a good +time to fix the underlying problem in the packages, because pip will +be stricter from here on out. + + +How to test +----------- + +1. **Install the beta** [specific instructions TKTK]. + +2. **Run ``pip check`` on your current environment**. This + will report if you have any inconsistencies in your set of installed + packages. Having a clean installation will make it much less likely + that you will hit issues when the new resolver is released (and may + address hidden problems in your current environment!). If you run + ``pip check`` and run into stuff you can’t figure out, please `ask + for help in our issuetracker or chat `__. + +3. **Test the new version of pip** (see below). While we have tried to + make sure that pip’s test suite covers as many cases as we can, we + are very aware that there are people using pip with many different + workflows and build processes, and we will not be able to cover all + of those without your help. + + - If you use pip to install your software, try out the new resolver + and let us know if it works for you with ``pip install``. Try: + - installing several packages simultaneously + - re-creating an environment using a ``requirements.txt`` file + - If you have a build pipeline that depends on pip installing your + dependencies for you, check that the new resolver does what you + need. + - Run your project’s CI (test suite, build process, etc.) using the + new resolver, and let us know of any issues. + - If you have encountered resolver issues with pip in the past, + check whether the new resolver fixes them. Also, let us know if + the new resolver has issues with any workarounds you put in to + address the current resolver’s limitations. We’ll need to ensure + that people can transition off such workarounds smoothly. + - If you develop or support a tool that wraps pip or uses it to + deliver part of your functionality, please test your integration + with the beta. + +Please report bugs [GitHub link or something else TKTK]. + +Setups we might need more testing on +------------------------------------ + +* Windows + +* Macintosh + +* Debian, Fedora, Red Hat, CentOS, Mint, Arch, Raspbian, Gentoo + +* Japanese-localized filesystems/OSes + +* Multi-user installations + +* Using virtualenvs + +* Dependency resolution for any kind of version control systems (e.g., you are installing from Git, Subversion, Mercurial, or CVS) + +* Installing from source code held in local directories + +* Using the most recent versions of Python 3.6, 3.7, 3.8, and 3.9 + +* Customized terminals (where you have modified how error messages and standard output display) + + .. _freeze: https://pip.pypa.io/en/latest/reference/pip_freeze/ From d51fd4799cd3067da493c15f10ac8dad9f8ae6ea Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Wed, 24 Jun 2020 00:32:53 -0400 Subject: [PATCH 02/18] docs: include force-reinstall item --- docs/html/user_guide.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 96e3e9350..2c2b4725a 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1088,6 +1088,7 @@ How to test and let us know if it works for you with ``pip install``. Try: - installing several packages simultaneously - re-creating an environment using a ``requirements.txt`` file + - using ``pip --force-reinstall`` to check whether it does what you think it should - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need. From d8d6da6985586d2a1e921cb96fd0a642b8ef5445 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Wed, 24 Jun 2020 00:35:57 -0400 Subject: [PATCH 03/18] Docs: add item about constraints files --- docs/html/user_guide.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 2c2b4725a..fede33db6 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1064,6 +1064,12 @@ incompatible or inconsistent requirements combinations, now's a good time to fix the underlying problem in the packages, because pip will be stricter from here on out. +We are also changing our support for :ref:`Constraints Files` : + +* Unnamed requirements are not allowed as constraints +* Links are not allowed as constraints +* Constraints cannot have extras + How to test ----------- @@ -1089,6 +1095,7 @@ How to test - installing several packages simultaneously - re-creating an environment using a ``requirements.txt`` file - using ``pip --force-reinstall`` to check whether it does what you think it should + - using constraints files - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need. From 0783460a7e1f902c62192f9d78ac7466a98b1b5f Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Mon, 29 Jun 2020 23:05:14 -0400 Subject: [PATCH 04/18] Improve coverage of conflicts from installed packages --- docs/html/user_guide.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index fede33db6..2c074f601 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1031,7 +1031,7 @@ Please test pip 20.2b2. Watch out for ------------- -The big thing in this beta is changes to the pip dependency resolver within pip. +The big change in this beta is to the pip dependency resolver within pip. Computers need to know the right order to install pieces of software ("to install `x`, you need to install `y` first"). So, when Python @@ -1064,6 +1064,14 @@ incompatible or inconsistent requirements combinations, now's a good time to fix the underlying problem in the packages, because pip will be stricter from here on out. +This also means that, when you run a ``pip install`` command, pip only +considers the packages you are installing in that command, and may +break already-installed packages. It will not guarantee that your +environment will be consistent all the time. If you ``pip install x`` +and then ``pip install y``, it's possible that the version of ``y`` +you get will be different than it would be if you had run ``pip +install x y`` in a single command. + We are also changing our support for :ref:`Constraints Files` : * Unnamed requirements are not allowed as constraints From 5bf4c5637681a67f2d415b29de7847648afe4b3c Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Thu, 23 Jul 2020 13:12:43 -0400 Subject: [PATCH 05/18] WIP: update info about release plan --- docs/html/user_guide.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 2c074f601..63200b632 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1023,15 +1023,22 @@ of ability. Some examples that you could consider include: * ``distlib`` - Packaging and distribution utilities (including functions for interacting with PyPI). -Testing the 20.2 beta -===================== +Changes to the pip dependency resolver in 20.2 +============================================== -Please test pip 20.2b2. +pip 20.1 included an alpha version of the new resolver (hidden behind +an optional ``--unstable-feature=resolver`` flag). pip 20.2 includes a +robust beta of the new resolver (hidden behind an optional +``--use-feature=2020-resolver`` flag) that we encourage you to +test. We will continue to improve the pip dependency resolver in +response to testers' feedback. This will help us prepare to release +pip 20.3, with the new resolver on by default, in October. Watch out for ------------- -The big change in this beta is to the pip dependency resolver within pip. +The big change in this release is to the pip dependency resolver +within pip. Computers need to know the right order to install pieces of software ("to install `x`, you need to install `y` first"). So, when Python @@ -1082,7 +1089,7 @@ We are also changing our support for :ref:`Constraints Files` : How to test ----------- -1. **Install the beta** [specific instructions TKTK]. +1. **Install pip 20.2** with ``python -m pip install --upgrade pip``. 2. **Run ``pip check`` on your current environment**. This will report if you have any inconsistencies in your set of installed @@ -1116,7 +1123,7 @@ How to test that people can transition off such workarounds smoothly. - If you develop or support a tool that wraps pip or uses it to deliver part of your functionality, please test your integration - with the beta. + with pip 20.2. Please report bugs [GitHub link or something else TKTK]. From 2142d04c722dae69d4dce1ab9b7165aa91fb8445 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:05:15 -0400 Subject: [PATCH 06/18] WIP: improvements to list of things to test --- docs/html/user_guide.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 63200b632..c79c11d3a 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1023,8 +1023,8 @@ of ability. Some examples that you could consider include: * ``distlib`` - Packaging and distribution utilities (including functions for interacting with PyPI). -Changes to the pip dependency resolver in 20.2 -============================================== +Changes to the pip dependency resolver in 20.2 (2020) +===================================================== pip 20.1 included an alpha version of the new resolver (hidden behind an optional ``--unstable-feature=resolver`` flag). pip 20.2 includes a @@ -1130,19 +1130,23 @@ Please report bugs [GitHub link or something else TKTK]. Setups we might need more testing on ------------------------------------ -* Windows +* Windows, including Windows Subsystem for Linux (WSL) * Macintosh * Debian, Fedora, Red Hat, CentOS, Mint, Arch, Raspbian, Gentoo -* Japanese-localized filesystems/OSes +* non-Latin localized filesystems and OSes, such as Japanese, Chinese, and Korean, and right-to-left such as Hebrew, Urdu, and Arabic * Multi-user installations -* Using virtualenvs +* Requirements files with 100+ packages -* Dependency resolution for any kind of version control systems (e.g., you are installing from Git, Subversion, Mercurial, or CVS) +* Requirements files that include hashes or pinned dependencies (perhaps as output from ``pip-compile`` or ``pip-tools``) + +* Continuous integration/continuous deployment setups + +* Installing from any kind of version control systems (i.e., Git, Subversion, Mercurial, or CVS) * Installing from source code held in local directories From 0ec91a541f1db22581c755419851b2abb833dffb Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:23:07 -0400 Subject: [PATCH 07/18] WIP: moving test suggestions from other issue Related to #8099. --- docs/html/user_guide.rst | 82 +++++++++++++++++++++++++++++++++++----- 1 file changed, 73 insertions(+), 9 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index c79c11d3a..16a94efc4 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1031,8 +1031,9 @@ an optional ``--unstable-feature=resolver`` flag). pip 20.2 includes a robust beta of the new resolver (hidden behind an optional ``--use-feature=2020-resolver`` flag) that we encourage you to test. We will continue to improve the pip dependency resolver in -response to testers' feedback. This will help us prepare to release -pip 20.3, with the new resolver on by default, in October. +response to testers' feedback. Please give us feedback through the +`resolver testing survey`_. This will help us prepare to release pip +20.3, with the new resolver on by default, in October. Watch out for ------------- @@ -1099,11 +1100,20 @@ How to test ``pip check`` and run into stuff you can’t figure out, please `ask for help in our issuetracker or chat `__. -3. **Test the new version of pip** (see below). While we have tried to - make sure that pip’s test suite covers as many cases as we can, we - are very aware that there are people using pip with many different - workflows and build processes, and we will not be able to cover all - of those without your help. +3. **Test the new version of pip** (see below). To test the new + resolver, use the ``--use-feature=2020-resolver`` flag, as in: + + ``pip install example --use-feature=2020-resolver`` + + The more feedback we can get, the more we can make sure that the + final release is solid. (Only try the new resolver **in a + non-production environment**, though - it isn't ready for you to + rely on in production!) + + While we have tried to make sure that pip’s test suite covers as + many cases as we can, we are very aware that there are people using + pip with many different workflows and build processes, and we will + not be able to cover all of those without your help. - If you use pip to install your software, try out the new resolver and let us know if it works for you with ``pip install``. Try: @@ -1125,7 +1135,7 @@ How to test deliver part of your functionality, please test your integration with pip 20.2. -Please report bugs [GitHub link or something else TKTK]. +4. **Please report bugs** through the `resolver testing survey`_. Setups we might need more testing on ------------------------------------ @@ -1142,17 +1152,71 @@ Setups we might need more testing on * Requirements files with 100+ packages +* Installing a package that has multiple requirements files + * Requirements files that include hashes or pinned dependencies (perhaps as output from ``pip-compile`` or ``pip-tools``) * Continuous integration/continuous deployment setups -* Installing from any kind of version control systems (i.e., Git, Subversion, Mercurial, or CVS) +* Installing from any kind of version control systems (i.e., Git, Subversion, Mercurial, or CVS), per :ref:`VCS Support` * Installing from source code held in local directories * Using the most recent versions of Python 3.6, 3.7, 3.8, and 3.9 +* PyPy + * Customized terminals (where you have modified how error messages and standard output display) +Examples to try +^^^^^^^^^^^^^^^ + +Install: + +* `tensorflow`_ +* ``hacking`` +* ``pycodestyle`` +* ``pandas`` +* ``tablib`` +* ``elasticsearch`` and ``requests`` together +* ``six`` and ``cherrypy`` together +* ``pip install flake8-import-order==0.17.1 flake8==3.5.0 --use-feature=2020-resolver`` +* ``pip install tornado==5.0 sprockets.http==1.5.0 --use-feature=2020-resolver`` + +Try: + +* ``pip install`` +* ``pip uninstall`` +* ``pip check`` +* ``pip cache`` + + +Tell us about +------------- + +Specific things we'd love to get feedback on: + +* Cases where the new resolver produces the wrong result, + obviously. We hope there won't be too many of these, but we'd like + to trap such bugs now. + +* Cases where the resolver produced an error when you believe it + should have been able to work out what to do. + +* Cases where the resolver gives an error because there's a problem + with your requirements, but you need better information to work out + what's wrong. + +* If you have workarounds to address issues with the current resolver, + does the new resolver let you remove those workarounds? Tell us! + + +Context and followup +-------------------- + +As discussed in `our announcement on the PSF blog`_, the pip team are in the process of developing a new "dependency resolver" (the part of pip that works out what to install based on your requirements). We have reached a major milestone in this work, and have a testable ("beta") version of the resolver, which you can test, included in this release. .. _freeze: https://pip.pypa.io/en/latest/reference/pip_freeze/ +.. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en +.. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html +.. _tensorflow: https://pypi.org/project/tensorflow/ From af0633e63300e58453f1ce73106144a003198c67 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:29:23 -0400 Subject: [PATCH 08/18] WIP formatting and links --- docs/html/user_guide.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 16a94efc4..13c0f8d27 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1092,7 +1092,7 @@ How to test 1. **Install pip 20.2** with ``python -m pip install --upgrade pip``. -2. **Run ``pip check`` on your current environment**. This +2. **Validate your current environment** by running ``pip check``. This will report if you have any inconsistencies in your set of installed packages. Having a clean installation will make it much less likely that you will hit issues when the new resolver is released (and may @@ -1154,7 +1154,11 @@ Setups we might need more testing on * Installing a package that has multiple requirements files -* Requirements files that include hashes or pinned dependencies (perhaps as output from ``pip-compile`` or ``pip-tools``) +* Requirements files that include hashes (:ref:`hash-checking mode`) + or pinned dependencies (perhaps as output from ``pip-compile`` or + ``pip-tools``) + +* Using :ref:`Constraints Files` * Continuous integration/continuous deployment setups From 483dec270a1d7c0e593f055800d74c8f1b59c69f Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:33:31 -0400 Subject: [PATCH 09/18] WIP formatting --- docs/html/user_guide.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 13c0f8d27..6a6fd8151 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1080,7 +1080,7 @@ and then ``pip install y``, it's possible that the version of ``y`` you get will be different than it would be if you had run ``pip install x y`` in a single command. -We are also changing our support for :ref:`Constraints Files` : +We are also changing our support for :ref:`Constraints Files`: * Unnamed requirements are not allowed as constraints * Links are not allowed as constraints @@ -1098,7 +1098,7 @@ How to test that you will hit issues when the new resolver is released (and may address hidden problems in your current environment!). If you run ``pip check`` and run into stuff you can’t figure out, please `ask - for help in our issuetracker or chat `__. + for help in our issue tracker or chat `__. 3. **Test the new version of pip** (see below). To test the new resolver, use the ``--use-feature=2020-resolver`` flag, as in: @@ -1119,7 +1119,8 @@ How to test and let us know if it works for you with ``pip install``. Try: - installing several packages simultaneously - re-creating an environment using a ``requirements.txt`` file - - using ``pip --force-reinstall`` to check whether it does what you think it should + - using ``pip install --force-reinstall`` to check whether + it does what you think it should - using constraints files - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you @@ -1155,7 +1156,7 @@ Setups we might need more testing on * Installing a package that has multiple requirements files * Requirements files that include hashes (:ref:`hash-checking mode`) - or pinned dependencies (perhaps as output from ``pip-compile`` or + or pinned dependencies (perhaps as output from ``pip-compile`` within ``pip-tools``) * Using :ref:`Constraints Files` From e898267d94c5e7c9e41cdb1cf86e924818bbb3ba Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:35:14 -0400 Subject: [PATCH 10/18] WIP fixing tabs spaces problem --- docs/html/user_guide.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 6a6fd8151..63f54d1fa 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1119,9 +1119,9 @@ How to test and let us know if it works for you with ``pip install``. Try: - installing several packages simultaneously - re-creating an environment using a ``requirements.txt`` file - - using ``pip install --force-reinstall`` to check whether - it does what you think it should - - using constraints files + - using ``pip install --force-reinstall`` to check whether + it does what you think it should + - using constraints files - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need. From a910ef6d37ce98906964c1c542121adecb4378a7 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:43:07 -0400 Subject: [PATCH 11/18] WIP linking to issues Addresses #6536 and #6628. --- docs/html/user_guide.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 63f54d1fa..14d5a966b 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1082,7 +1082,7 @@ install x y`` in a single command. We are also changing our support for :ref:`Constraints Files`: -* Unnamed requirements are not allowed as constraints +* Unnamed requirements are not allowed as constraints (see `#6628`_) * Links are not allowed as constraints * Constraints cannot have extras @@ -1225,3 +1225,4 @@ As discussed in `our announcement on the PSF blog`_, the pip team are in the pro .. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _tensorflow: https://pypi.org/project/tensorflow/ +.. _#6628: https://github.com/pypa/pip/issues/6628 From 6bfbce0adcfee88d313e942878287524888b4d36 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:47:07 -0400 Subject: [PATCH 12/18] WIP adding links to issues Addresses #8210 , #8253 --- docs/html/user_guide.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 14d5a966b..af508864e 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1082,9 +1082,9 @@ install x y`` in a single command. We are also changing our support for :ref:`Constraints Files`: -* Unnamed requirements are not allowed as constraints (see `#6628`_) -* Links are not allowed as constraints -* Constraints cannot have extras +* Unnamed requirements are not allowed as constraints (see `#6628`_ and `#8210`_) +* Links are not allowed as constraints (see `#8253`_) +* Constraints cannot have extras (see `#6628`_) How to test @@ -1226,3 +1226,5 @@ As discussed in `our announcement on the PSF blog`_, the pip team are in the pro .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _tensorflow: https://pypi.org/project/tensorflow/ .. _#6628: https://github.com/pypa/pip/issues/6628 +.. _#8210: https://github.com/pypa/pip/issues/8210 +.. _#8253: https://github.com/pypa/pip/issues/8253 From 223e37945612be82fbcf4ba9a51721419b1073a5 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 09:50:46 -0400 Subject: [PATCH 13/18] WIP formatting --- docs/html/user_guide.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index af508864e..2330d04c7 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1082,9 +1082,9 @@ install x y`` in a single command. We are also changing our support for :ref:`Constraints Files`: -* Unnamed requirements are not allowed as constraints (see `#6628`_ and `#8210`_) -* Links are not allowed as constraints (see `#8253`_) -* Constraints cannot have extras (see `#6628`_) +* Unnamed requirements are not allowed as constraints (see :issue:`6628` and :issue:`8210`) +* Links are not allowed as constraints (see :issue:`8253`) +* Constraints cannot have extras (see :issue:`6628`) How to test @@ -1225,6 +1225,3 @@ As discussed in `our announcement on the PSF blog`_, the pip team are in the pro .. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _tensorflow: https://pypi.org/project/tensorflow/ -.. _#6628: https://github.com/pypa/pip/issues/6628 -.. _#8210: https://github.com/pypa/pip/issues/8210 -.. _#8253: https://github.com/pypa/pip/issues/8253 From 1e5e544b5c760aa5fb9283e0dcc29f70dcbc7b76 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 10:00:40 -0400 Subject: [PATCH 14/18] Ref links and followup tips --- docs/html/development/release-process.rst | 1 + docs/html/user_guide.rst | 14 +++++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/html/development/release-process.rst b/docs/html/development/release-process.rst index 8ccc53403..cbfbce4ad 100644 --- a/docs/html/development/release-process.rst +++ b/docs/html/development/release-process.rst @@ -30,6 +30,7 @@ to need extra work before being released, the release manager always has the option to back out the partial change prior to a release. The PR can then be reworked and resubmitted for the next release. +.. _`Deprecation Policy`: Deprecation Policy ================== diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 2330d04c7..ad95050af 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1023,6 +1023,8 @@ of ability. Some examples that you could consider include: * ``distlib`` - Packaging and distribution utilities (including functions for interacting with PyPI). +.. _`Resolver changes 2020`: + Changes to the pip dependency resolver in 20.2 (2020) ===================================================== @@ -1215,13 +1217,23 @@ Specific things we'd love to get feedback on: * If you have workarounds to address issues with the current resolver, does the new resolver let you remove those workarounds? Tell us! +Please let us know through the `resolver testing survey`_. Context and followup -------------------- -As discussed in `our announcement on the PSF blog`_, the pip team are in the process of developing a new "dependency resolver" (the part of pip that works out what to install based on your requirements). We have reached a major milestone in this work, and have a testable ("beta") version of the resolver, which you can test, included in this release. +As discussed in `our announcement on the PSF blog`_, the pip team are +in the process of developing a new "dependency resolver" (the part of +pip that works out what to install based on your requirements). Since +this work will not change user-visible behavior described in the pip +documentation, this change is not covered by the :ref:`Deprecation +Policy`. + +We're tracking our rollout in :issue:`6536` and you can watch for +announcements on the `low-traffic packaging announcements list`_. .. _freeze: https://pip.pypa.io/en/latest/reference/pip_freeze/ .. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _tensorflow: https://pypi.org/project/tensorflow/ +.. _low-traffic packaging announcements list: https://mail.python.org/mailman3/lists/pypi-announce.python.org/ From d374b0d87754545d761957924b319f67d856cc3b Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 10:05:11 -0400 Subject: [PATCH 15/18] Add changelog entry --- news/6536.feature | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 news/6536.feature diff --git a/news/6536.feature b/news/6536.feature new file mode 100644 index 000000000..d63957914 --- /dev/null +++ b/news/6536.feature @@ -0,0 +1,11 @@ +Add a beta version of pip's next-generation dependency resolver. + +Move pip's new resolver into beta, remove the +``--unstable-feature=resolver`` flag, and enable the +``--use-feature=2020-resolver`` flag. The new resolver is +significantly stricter and more consistent when it receives +incompatible instructions, and reduces support for certain kinds of +:ref:`Constraints Files`, so some workarounds and workflows may +break. More details about how to test and migrate, and how to report +issues, at :ref:`Resolver changes 2020` . Maintainers are preparing to +release pip 20.3, with the new resolver on by default, in October. From ec736200f4d97e14c368535df7482e8e645e05d3 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 10:16:58 -0400 Subject: [PATCH 16/18] formatting --- docs/html/user_guide.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index ad95050af..f7a022c42 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1054,20 +1054,20 @@ logic, and easier for you to use and troubleshoot. The most significant changes to the resolver are: * It will **reduce inconsistency**: it will *no longer install a -combination of packages that is mutually inconsistent*. In older -versions of pip, it is possible for pip to install a package which -does not satisfy the declared requirements of another installed -package. For example, in pip 20.0, ``pip install "six<1.12" -"virtualenv==20.0.2"`` does the wrong thing, “successfully” installing -``six==1.11``, even though ``virtualenv==20.0.2`` requires -``six>=1.12.0,<2`` (`defined here -`__). -The new resolver, instead, outright rejects installing anything -if it gets that input. + combination of packages that is mutually inconsistent*. In older + versions of pip, it is possible for pip to install a package which + does not satisfy the declared requirements of another installed + package. For example, in pip 20.0, ``pip install "six<1.12" + "virtualenv==20.0.2"`` does the wrong thing, “successfully” installing + ``six==1.11``, even though ``virtualenv==20.0.2`` requires + ``six>=1.12.0,<2`` (`defined here + `__). + The new resolver, instead, outright rejects installing anything if it + gets that input. * It will be **stricter** - if you ask pip to install two packages with -incompatible requirements, it will refuse (rather than installing a -broken combination, like it did in previous versions). + incompatible requirements, it will refuse (rather than installing a + broken combination, like it did in previous versions). So, if you have been using workarounds to force pip to deal with incompatible or inconsistent requirements combinations, now's a good From 263f85b0add0b7d5f170ef6acd03bde87a09df1e Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 10:21:52 -0400 Subject: [PATCH 17/18] formatting fix --- docs/html/user_guide.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index f7a022c42..9ef6dfffc 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1119,11 +1119,13 @@ How to test - If you use pip to install your software, try out the new resolver and let us know if it works for you with ``pip install``. Try: + - installing several packages simultaneously - re-creating an environment using a ``requirements.txt`` file - using ``pip install --force-reinstall`` to check whether it does what you think it should - using constraints files + - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need. From 66d39f942ce0c2b7c9ba1f2d10dac4e1ffd59904 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Tue, 28 Jul 2020 11:22:57 -0400 Subject: [PATCH 18/18] Clarify on conflicts and multiplicity --- docs/html/user_guide.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 9ef6dfffc..702a97d0c 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1049,7 +1049,7 @@ programmers share software as packages, they have to precisely describe those installation prerequisites, and pip needs to navigate tricky situations where it's getting conflicting instructions. This new dependency resolver will make pip better at handling that tricky -logic, and easier for you to use and troubleshoot. +logic, and make pip easier for you to use and troubleshoot. The most significant changes to the resolver are: @@ -1080,7 +1080,9 @@ break already-installed packages. It will not guarantee that your environment will be consistent all the time. If you ``pip install x`` and then ``pip install y``, it's possible that the version of ``y`` you get will be different than it would be if you had run ``pip -install x y`` in a single command. +install x y`` in a single command. We would like your thoughts on what +pip's behavior should be; please answer `our survey on upgrades that +create conflicts`_. We are also changing our support for :ref:`Constraints Files`: @@ -1157,7 +1159,7 @@ Setups we might need more testing on * Requirements files with 100+ packages -* Installing a package that has multiple requirements files +* An installation workflow that involves multiple requirements files * Requirements files that include hashes (:ref:`hash-checking mode`) or pinned dependencies (perhaps as output from ``pip-compile`` within @@ -1239,3 +1241,4 @@ announcements on the `low-traffic packaging announcements list`_. .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _tensorflow: https://pypi.org/project/tensorflow/ .. _low-traffic packaging announcements list: https://mail.python.org/mailman3/lists/pypi-announce.python.org/ +.. _our survey on upgrades that create conflicts: https://docs.google.com/forms/d/e/1FAIpQLSeBkbhuIlSofXqCyhi3kGkLmtrpPOEBwr6iJA6SzHdxWKfqdA/viewform