From 3a20399d938b55b580b169f7fae6037646e8d8e6 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Fri, 23 Oct 2020 17:58:53 -0400 Subject: [PATCH 1/4] Update user guide to change resolver default In pip 20.3, the new resolver will be the default. Therefore, this commit updates user docs to change instructions for testing, migration, etc. Towards #8937. --- README.rst | 5 +- .../html/development/architecture/anatomy.rst | 3 +- .../architecture/upgrade-options.rst | 4 + docs/html/development/release-process.rst | 2 + docs/html/index.rst | 2 +- docs/html/user_guide.rst | 135 +++++++++--------- news/9044.doc.rst | 1 + 7 files changed, 83 insertions(+), 69 deletions(-) create mode 100644 news/9044.doc.rst diff --git a/README.rst b/README.rst index 395b642d6..3d5bd0f29 100644 --- a/README.rst +++ b/README.rst @@ -19,7 +19,7 @@ We release updates regularly, with a new version every 3 months. Find more detai * `Release notes`_ * `Release process`_ -In 2020, we're working on improvements to the heart of pip. Please `learn more and take our survey`_ to help us do it right. +In pip 20.3, we're making a big improvement to the heart of pip. Please `learn more`_ and `take our survey`_ to help us do it right. If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms: @@ -48,7 +48,8 @@ rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_. .. _Release process: https://pip.pypa.io/en/latest/development/release-process/ .. _GitHub page: https://github.com/pypa/pip .. _Development documentation: https://pip.pypa.io/en/latest/development -.. _learn more and take our survey: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html +.. _learn more: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020 +.. _take our survey: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _Issue tracking: https://github.com/pypa/pip/issues .. _Discourse channel: https://discuss.python.org/c/packaging .. _Development mailing list: https://mail.python.org/mailman3/lists/distutils-sig.python.org/ diff --git a/docs/html/development/architecture/anatomy.rst b/docs/html/development/architecture/anatomy.rst index 4b117bafe..46bba4489 100644 --- a/docs/html/development/architecture/anatomy.rst +++ b/docs/html/development/architecture/anatomy.rst @@ -97,8 +97,7 @@ Within ``src/``: * ``pep425tags.py`` -- getting refactored into packaging.tags (a library on PyPI) which is external to pip (but vendored by pip). :pep:`425` tags: turns out lots of people want this! Compatibility tags for built distributions -> e.g., platform, Python version, etc. * ``pyproject.py`` -- ``pyproject.toml`` is a new standard (:pep:`518` and :pep:`517`). This file reads pyproject.toml and passes that info elsewhere. The rest of the processing happens in a different file. All the handling for 517 and 518 is in a different file. - * ``req/`` *[*\ **A DIRECTORY THAT NEEDS REFACTORING. A LOT**\ *\ …… Remember Step 3? Dependency resolution etc.? This is that step! Each file represents … have the entire flow of installing & uninstalling, getting info about packages…. Some files here are more than 1,000 lines long! (used to be longer?!) Refactor will deeply improve developer experience.]* - * ``resolve.py`` -- This is where the current dependency resolution algorithm sits. Pradyun is `improving the pip dependency resolver`_. Pradyun will get rid of this file and replace it with a directory called “resolution”. (this work is in git master…. There is further work that is going to be in a branch soon) + * ``req/`` *[*\ **A DIRECTORY THAT NEEDS REFACTORING. A LOT**\ *\ …… Remember Step 3? Dependency resolution etc.? This is that step! Each file represents … have the entire flow of installing & uninstalling, getting info about packages…. Some files here are more than 1,000 lines long! (used to be longer?!) Refactor will deeply improve developer experience. Also, we're `improving the pip dependency resolver`_ in 2020 so a bunch of this is changing.]* * ``utils/`` *[everything that is not “operationally” pip ….. Misc functions and files get dumped. There’s some organization here. There’s a models.py here which needs refactoring. Deprecation.py is useful, as are other things, but some things do not belong here. There ought to be some GitHub issues for refactoring some things here. Maybe a few issues with checkbox lists.]* * ``vcs/`` *[stands for Version Control System. Where pip handles all version control stuff -- one of the ``pip install`` arguments you can use is a version control link. Are any of these commands vendored? No, via subprocesses. For performance, it makes sense (we think) to do this instead of pygitlib2 or similar -- and has to be pure Python, can’t include C libraries, because you can’t include compiled C stuff, because you might not have it for the platform you are running on.]* diff --git a/docs/html/development/architecture/upgrade-options.rst b/docs/html/development/architecture/upgrade-options.rst index c87e6c976..36e34c71e 100644 --- a/docs/html/development/architecture/upgrade-options.rst +++ b/docs/html/development/architecture/upgrade-options.rst @@ -6,6 +6,10 @@ When installing packages, pip chooses a distribution file, and installs it in the user's environment. There are many choices involved in deciding which file to install, and these are controlled by a variety of options. +.. note:: + + This section of the documentation needs to be updated per + :ref:`Resolver changes 2020`. Controlling what gets installed =============================== diff --git a/docs/html/development/release-process.rst b/docs/html/development/release-process.rst index 1c6889ef0..e8c4f5795 100644 --- a/docs/html/development/release-process.rst +++ b/docs/html/development/release-process.rst @@ -65,6 +65,8 @@ their merits. ``pip._internal.utils.deprecation.deprecated``. The function is not a part of pip's public API. +.. _`Python 2 Support`: + Python 2 Support ---------------- diff --git a/docs/html/index.rst b/docs/html/index.rst index 64fc34c9d..62217b368 100644 --- a/docs/html/index.rst +++ b/docs/html/index.rst @@ -17,7 +17,7 @@ Please take a look at our documentation for how to install and use pip: ux_research_design news -In 2020, we're working on improvements to the heart of pip: :ref:`Resolver changes 2020`. Please `learn more and take our survey`_ to help us do it right, and `join our user experience surveys pool`_. +In pip 20.3, we're making a big improvement to the heart of pip: :ref:`Resolver changes 2020`. Please `learn more and take our survey`_ to help us do it right, and `join our user experience surveys pool`_. If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms: diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index 0e0f6cd3e..c6ad72f7e 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -197,16 +197,18 @@ In practice, there are 4 common uses of Requirements files: py -m pip freeze > requirements.txt py -m pip install -r requirements.txt -2. Requirements files are used to force pip to properly resolve dependencies. - As it is now, pip `doesn't have true dependency resolution - `_, but instead simply uses the first - specification it finds for a project. E.g. if ``pkg1`` requires - ``pkg3>=1.0`` and ``pkg2`` requires ``pkg3>=1.0,<=2.0``, and if ``pkg1`` is - resolved first, pip will only use ``pkg3>=1.0``, and could easily end up - installing a version of ``pkg3`` that conflicts with the needs of ``pkg2``. - To solve this problem, you can place ``pkg3>=1.0,<=2.0`` (i.e. the correct - specification) into your requirements file directly along with the other top - level requirements. Like so:: +2. Requirements files are used to force pip to properly resolve + dependencies. In versions of pip prior to 20.3, pip `doesn't have + true dependency resolution + `_, but instead simply uses + the first specification it finds for a project. E.g. if ``pkg1`` + requires ``pkg3>=1.0`` and ``pkg2`` requires ``pkg3>=1.0,<=2.0``, + and if ``pkg1`` is resolved first, pip will only use ``pkg3>=1.0``, + and could easily end up installing a version of ``pkg3`` that + conflicts with the needs of ``pkg2``. To solve this problem for + pip 20.2 and previous, you can place ``pkg3>=1.0,<=2.0`` (i.e. the + correct specification) into your requirements file directly along + with the other top level requirements. Like so:: pkg1 pkg2 @@ -1095,6 +1097,8 @@ archives are built with identical packages. to use such a package, see :ref:`Controlling setup_requires`. +.. _`Fixing conflicting dependencies`: + Fixing conflicting dependencies =============================== @@ -1103,8 +1107,10 @@ pip users who encounter an error where pip cannot install their specified packages due to conflicting dependencies (a ``ResolutionImpossible`` error). -This documentation is specific to the new resolver, which you can use -with the flag ``--use-feature=2020-resolver``. +This documentation is specific to the new resolver, which is the +default behavior in pip 20.3 and later. If you are using pip 20.2, you +can invoke the new resolver by using the flag +``--use-feature=2020-resolver``. Understanding your error message -------------------------------- @@ -1376,17 +1382,20 @@ of ability. Some examples that you could consider include: .. _`Resolver changes 2020`: -Changes to the pip dependency resolver in 20.2 (2020) +Changes to the pip dependency resolver in 20.3 (2020) ===================================================== -pip 20.1 included an alpha version of the new resolver (hidden behind -an optional ``--unstable-feature=resolver`` flag). pip 20.2 removes -that flag, and includes a robust beta of the new resolver (hidden -behind an optional ``--use-feature=2020-resolver`` flag) that we -encourage you to test. We will continue to improve the pip dependency -resolver in response to testers' feedback. Please give us feedback -through the `resolver testing survey`_. This will help us prepare to -release pip 20.3, with the new resolver on by default, in October. +pip 20.3 has a new dependency resolver, on by default. (pip 20.1 and +20.2 included pre-release versions of the new dependency resolver, +hidden behind optional user flags.) Read below for a migration guide, +how to invoke the legacy resolver, and the deprecation timeline. We +also made a `two-minute video explanation`_ you can watch. + +We will continue to improve the pip dependency resolver in response to +testers' feedback. Please give us feedback through the `resolver +testing survey`_. + +.. _`Migration guide for 2020 resolver changes`: Watch out for ------------- @@ -1442,28 +1451,20 @@ We are also changing our support for :ref:`Constraints Files`: * Constraints cannot have extras (see :issue:`6628`) -How to test ------------ +How to upgrade and migrate +-------------------------- -1. **Install pip 20.2** with ``python -m pip install --upgrade pip``. +1. **Install pip 20.3** with ``python -m pip install --upgrade pip``. 2. **Validate your current environment** by running ``pip check``. This will report if you have any inconsistencies in your set of installed packages. Having a clean installation will make it much less likely - that you will hit issues when the new resolver is released (and may + that you will hit issues with the new resolver (and may address hidden problems in your current environment!). If you run ``pip check`` and run into stuff you can’t figure out, please `ask for help in our issue tracker or chat `__. -3. **Test the new version of pip** (see below). To test the new - resolver, use the ``--use-feature=2020-resolver`` flag, as in: - - ``pip install example --use-feature=2020-resolver`` - - The more feedback we can get, the more we can make sure that the - final release is solid. (Only try the new resolver **in a - non-production environment**, though - it isn't ready for you to - rely on in production!) +3. **Test the new version of pip**. While we have tried to make sure that pip’s test suite covers as many cases as we can, we are very aware that there are people using @@ -1478,45 +1479,45 @@ How to test - using ``pip install --force-reinstall`` to check whether it does what you think it should - using constraints files + - the "Setups to test with special attention" and "Examples to try" below - If you have a build pipeline that depends on pip installing your dependencies for you, check that the new resolver does what you need. - - If you'd like pip to default to using the new resolver, run ``pip - config set global.use-feature 2020-resolver`` (for more on that - and the alternate ``PIP_USE_FEATURE`` environment variable - option, see `issue 8661`_). - - Run your project’s CI (test suite, build process, etc.) using the new resolver, and let us know of any issues. - If you have encountered resolver issues with pip in the past, - check whether the new resolver fixes them. Also, let us know if - the new resolver has issues with any workarounds you put in to - address the current resolver’s limitations. We’ll need to ensure - that people can transition off such workarounds smoothly. + check whether the new resolver fixes them, and read :ref:`Fixing + conflicting dependencies`. Also, let us know if the new resolver + has issues with any workarounds you put in to address the + current resolver’s limitations. We’ll need to ensure that people + can transition off such workarounds smoothly. - If you develop or support a tool that wraps pip or uses it to deliver part of your functionality, please test your integration - with pip 20.2. + with pip 20.3. -4. **Please report bugs** through the `resolver testing survey`_. +4. **Temporarily use the old resolver when necessary.** If you run + into resolution errors and need a workaround while you're fixing + their root causes, you can choose the old resolver behavior + using the flag ``--use-deprecated=legacy-resolver``. -Setups we might need more testing on ------------------------------------- + Per our :ref:`Python 2 Support` policy, pip 20.3 users who are + using Python 2 and who have trouble with the new resolver can + choose to switch to the old resolver behavior using the flag + ``--use-deprecated=legacy-resolver``. Python 2 users should + upgrade to Python 3 as soon as possible, since in pip 21.0 in + January 2021, pip will drop support for Python 2 altogether. -* Windows, including Windows Subsystem for Linux (WSL) +5. **Please report bugs** through the `resolver testing survey`_. -* Macintosh -* Debian, Fedora, Red Hat, CentOS, Mint, Arch, Raspbian, Gentoo - -* non-Latin localized filesystems and OSes, such as Japanese, Chinese, and Korean, and right-to-left such as Hebrew, Urdu, and Arabic - -* Multi-user installations +Setups to test with special attention +------------------------------------- * Requirements files with 100+ packages -* An installation workflow that involves multiple requirements files +* Installation workflows that involve multiple requirements files * Requirements files that include hashes (:ref:`hash-checking mode`) or pinned dependencies (perhaps as output from ``pip-compile`` within @@ -1530,12 +1531,6 @@ Setups we might need more testing on * Installing from source code held in local directories -* Using the most recent versions of Python 3.6, 3.7, 3.8, and 3.9 - -* PyPy - -* Customized terminals (where you have modified how error messages and standard output display) - Examples to try ^^^^^^^^^^^^^^^ @@ -1580,15 +1575,26 @@ Specific things we'd love to get feedback on: Please let us know through the `resolver testing survey`_. +.. _`Deprecation timeline for 2020 resolver changes`: + Deprecation timeline -------------------- We plan for the resolver changeover to proceed as follows, using :ref:`Feature Flags` and following our :ref:`Release Cadence`: -* pip 20.2: a beta of the new resolver is available, opt-in, using - the flag ``--use-feature=2020-resolver``. pip defaults to - legacy behavior. +* pip 20.1: an alpha version of the new resolver was available, + opt-in, using the optional flag + ``--unstable-feature=resolver``. pip defaulted to legacy + behavior. + +* pip 20.2: a beta of the new resolver was available, opt-in, using + the flag ``--use-feature=2020-resolver``. pip defaulted to legacy + behavior. Users of pip 20.2 who want pip to default to using the + new resolver can run ``pip config set global.use-feature + 2020-resolver`` (for more on that and the alternate + ``PIP_USE_FEATURE`` environment variable option, see `issue + 8661`_). * pip 20.3: pip defaults to the new resolver, but a user can opt-out and choose the old resolver behavior, using the flag @@ -1616,6 +1622,7 @@ announcements on the `low-traffic packaging announcements list`_ and .. _resolver testing survey: https://tools.simplysecure.org/survey/index.php?r=survey/index&sid=989272&lang=en .. _issue 8661: https://github.com/pypa/pip/issues/8661 .. _our announcement on the PSF blog: http://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html +.. _two-minute video explanation: https://www.youtube.com/watch?v=B4GQCBBsuNU .. _tensorflow: https://pypi.org/project/tensorflow/ .. _low-traffic packaging announcements list: https://mail.python.org/mailman3/lists/pypi-announce.python.org/ .. _our survey on upgrades that create conflicts: https://docs.google.com/forms/d/e/1FAIpQLSeBkbhuIlSofXqCyhi3kGkLmtrpPOEBwr6iJA6SzHdxWKfqdA/viewform diff --git a/news/9044.doc.rst b/news/9044.doc.rst new file mode 100644 index 000000000..58c7f39d2 --- /dev/null +++ b/news/9044.doc.rst @@ -0,0 +1 @@ +Update user docs to reflect new resolver as default in 20.3. From f4e96e95c95cbb84a250f90d8cedafe2bd534dc3 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Mon, 26 Oct 2020 15:47:02 -0400 Subject: [PATCH 2/4] Add redirect for heading in user guide --- docs/html/user_guide.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index c6ad72f7e..f6c4ee5dd 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1380,6 +1380,8 @@ of ability. Some examples that you could consider include: * ``distlib`` - Packaging and distribution utilities (including functions for interacting with PyPI). +.. _changes-to-the-pip-dependency-resolver-in-20-2-2020: + .. _`Resolver changes 2020`: Changes to the pip dependency resolver in 20.3 (2020) From 4ce9565aa1751d117e5190362c2000cf96c2e3c8 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Mon, 26 Oct 2020 16:01:24 -0400 Subject: [PATCH 3/4] Update docs about Python 2 and resolver Per #9019. --- docs/html/user_guide.rst | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/html/user_guide.rst b/docs/html/user_guide.rst index f6c4ee5dd..2e8bfeb02 100644 --- a/docs/html/user_guide.rst +++ b/docs/html/user_guide.rst @@ -1452,6 +1452,11 @@ We are also changing our support for :ref:`Constraints Files`: * Links are not allowed as constraints (see :issue:`8253`) * Constraints cannot have extras (see :issue:`6628`) +Per our :ref:`Python 2 Support` policy, pip 20.3 users who are using +Python 2 will use the legacy resolver by default. Python 2 users +should upgrade to Python 3 as soon as possible, since in pip 21.0 in +January 2021, pip will drop support for Python 2 altogether. + How to upgrade and migrate -------------------------- @@ -1504,13 +1509,6 @@ How to upgrade and migrate their root causes, you can choose the old resolver behavior using the flag ``--use-deprecated=legacy-resolver``. - Per our :ref:`Python 2 Support` policy, pip 20.3 users who are - using Python 2 and who have trouble with the new resolver can - choose to switch to the old resolver behavior using the flag - ``--use-deprecated=legacy-resolver``. Python 2 users should - upgrade to Python 3 as soon as possible, since in pip 21.0 in - January 2021, pip will drop support for Python 2 altogether. - 5. **Please report bugs** through the `resolver testing survey`_. From 7657c44409049043b8b7470baf0535849fb00800 Mon Sep 17 00:00:00 2001 From: Sumana Harihareswara Date: Wed, 28 Oct 2020 23:29:09 -0400 Subject: [PATCH 4/4] docs: Clarify links for resolver work and studies Co-Authored-By: Bernard Tyers --- README.rst | 5 +++-- docs/html/index.rst | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/README.rst b/README.rst index 3d5bd0f29..d8d12c850 100644 --- a/README.rst +++ b/README.rst @@ -19,7 +19,7 @@ We release updates regularly, with a new version every 3 months. Find more detai * `Release notes`_ * `Release process`_ -In pip 20.3, we're making a big improvement to the heart of pip. Please `learn more`_ and `take our survey`_ to help us do it right. +In pip 20.3, we're `making a big improvement to the heart of pip`_; `learn more`_. We want your input, so `sign up for our user experience research studies`_ to help us do it right. If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms: @@ -48,8 +48,9 @@ rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_. .. _Release process: https://pip.pypa.io/en/latest/development/release-process/ .. _GitHub page: https://github.com/pypa/pip .. _Development documentation: https://pip.pypa.io/en/latest/development +.. _making a big improvement to the heart of pip: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _learn more: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020 -.. _take our survey: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html +.. _sign up for our user experience research studies: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _Issue tracking: https://github.com/pypa/pip/issues .. _Discourse channel: https://discuss.python.org/c/packaging .. _Development mailing list: https://mail.python.org/mailman3/lists/distutils-sig.python.org/ diff --git a/docs/html/index.rst b/docs/html/index.rst index 62217b368..ce40b49fa 100644 --- a/docs/html/index.rst +++ b/docs/html/index.rst @@ -17,7 +17,7 @@ Please take a look at our documentation for how to install and use pip: ux_research_design news -In pip 20.3, we're making a big improvement to the heart of pip: :ref:`Resolver changes 2020`. Please `learn more and take our survey`_ to help us do it right, and `join our user experience surveys pool`_. +In pip 20.3, we're `making a big improvement to the heart of pip`_; :ref:`Resolver changes 2020`. We want your input, so `sign up for our user experience research studies`_ to help us do it right. If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms: @@ -40,8 +40,8 @@ rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_. .. _package installer: https://packaging.python.org/guides/tool-recommendations/ .. _Python Package Index: https://pypi.org -.. _join our user experience surveys pool: ux_research_design -.. _learn more and take our survey: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html +.. _making a big improvement to the heart of pip: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html +.. _sign up for our user experience research studies: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html .. _Installation: https://pip.pypa.io/en/stable/installing.html .. _Documentation: https://pip.pypa.io/en/stable/ .. _Changelog: https://pip.pypa.io/en/stable/news.html