This tremendously helps understand why a package is being fetched and
can help investigate and fix dependency resolver backtracking issues
when incoherent constraints/package sets are provided or when new
versions of a package trigger a completely different backtracking
strategy, leading to very hard to debug situations.
Use the same code to determine isolated environment paths at
dependency install time and at environment setup time. We do not care
about the exact paths but the paths needs to be consistent at package
installation time and environment setup.
This should fix issues observed on platforms that customize the
installation schemes, such as Debian and Homebrew, where dependency
installation and isolated build environment setup resolved to
different paths.
The PEP 668 expects an override mechanism to ease the transition.
This provides an override.
---------
Co-authored-by: Pradyun Gedam <pradyunsg@gmail.com>
Refactored `_get_index_url()` to get integration tests for the subprocess backend working.
Keyring support via the 'subprocess' provider can only retrieve a password, not a username-password combo. The username therefor MUST come from the URL.
If the URL obtained from the index does not contain a username then the username from a matching index is used. `_get_index_url()` does that matching.
The problem this refactoring solves is that the URL where a wheel or sdist can be downloaded from does not always start with the index url. Azure DevOps Artifacts Feeds are an example since it replaces the friendly name of the Feed with the GUID of the Feed. Causing `url.startswith(prefix)` to evaluate as `False`.
The new behaviour is to return the index which matches the netloc and has the longest common prefix of the `path` property of the value returned by `urllib.parse.urlsplit()`. The behaviour for resolving ties is unspecified.
The fix merged in 3b60e36289 also fixed
the wheel cache; this cherry-picks the test and news from the other pull
request to validate and document the fact.
This implements the PEP 668 logic to 'pip install' and 'pip uninstall'.
Are there any other commands that may need it?
This implementation disables the check is any of --prefix, --home, or
--target is provided, since those can indicate the command does not
actually install into the environment. Note that it is still possible
the command is still modifying the environment, but we don't have a
way to stop the user *that* determined to break the environment anyway
(they can always just use those flags in a virtual environment).
Also not sure how best this can be tested.
The scripts path was looked up passing explicitly the scheme to be
used using "nt" on Windows and "posix_prefix" everywhere else.
However, when the isolated build environment is created, packages are
installed using the default scheme for the platform. On most platforms
this works because normally "nt" and "posix_prefix" are the default
schemes.
However, Debian customizes sysconfig to use a "posix_local" scheme by
default and under this scheme the scripts path does not match the one
of the "posix_prefix" scheme. This results in scripts installed as
part of the build dependencies not to be found during the build, as
reported here https://github.com/mesonbuild/meson-python/issues/109
and here https://bugs.debian.org/1019293.
The problem can be solved omitting to specify a scheme when looking up
the scripts path. To future proof the path lookup, use the "venv"
scheme if available as done in #11598. For uniformity use similar
functions as used to lookup the library paths.
get_prefixed_libs() computes the Python path for libraries in a pip
isolation environment. Python 3.11 introduced the "venv" path scheme
to be used in these cases. Use it if available.
This solves a bug on Homebrew's Python 3.10 and later where the
default paths scheme when Python is invoked outside a virtual
environment is "osx_framework_library" and does not relative to the
"{base}" or "{platbase}" variables.
Fixes#11539.
Previously, the special case to generate 'pip' and 'easy_install' entry
points with the correct Python version (e.g. 'pip3.9' on Python 3.9)
only accounted for single-digit version segments, and did not work
correctly on Python 3.10 and up. This was missed when Python 3.10 was
released because we (accidentally) generated wheels that did not need
any such replacements, but was exposed in CPython 3.11.0 since it
bundled pip 22.3 generated against Python 3.10.
Git 2.38.1 patched CVE-2022-39253 by disaling automated fetch against a
file: repository. This breaks git submodule, which is used by a pip
test. Information on how projects relying on automated fetch should
configure git correctly after this change is lacking, so the test is
disabled for now until someone can come up with a better solution.
The tests still don't run without distutils
because they require virtualenv < 20 (and virtualenv 16 uses distutils),
but at least they don't import distutils directly now.
Fixes https://github.com/pypa/pip/issues/11521
shlex.split, used to split options in requirements.txt files, might
raise a ValueError when the input string is not well formed. Catch the
ValueError and re-raise it as a RequirementsFileParseError instead.
The new functionality serves a use case which was not previously possible with pip configuration files, namely the situation where you have a base Python installation and want to influence the pip configuration for all derivative virtual environments *without* changing the config for all other environments on a machine (global), or for all other environment run by the same user (user). Concretely, this could be used for a centrally managed network mounted filesystem based Python installation, from which multiple users can build virtual environments and inside which a specific pip configuration is needed (e.g. an index URL).
This patch adds a special check-job that produces a clear failure or
success status based on how the dependent jobs are doing. It is
possible to use it in GitHub's branch protection instead of having to
manually add and remove individual job names via the repo settings.
https://github.com/marketplace/actions/alls-green#why
use site.getsitepackages() where available
instead of just purelib/platlib,
which is often insufficient on e.g. System Pythons for Debian/macOS
handle virtualenv < 20 overwriting site.py without getsitepackages() by preserving current behavior.
Instead of creating a zip file from the current pip's sources, add the
current copy of pip, to the build environment's interpreter's import
system using `sys.meta_path`. This avoids the overhead of creating the
zipfile, allows us to use the current pip's sources as-is,
meaningfully reduces the size of the build environment and
speeds up the creation of the build environment.
This applies to the new importlib.metadata backend. The legacy
pkg_resources backend already does this (albeit accidentally).
A package inside a wheel is not guaranteed to "work" when directly
imported, so we should not treat it as an installed distribution.
The code to do this already exists in `get_csv_rows_for_installed`, but it's
broken due to inconsistent usage of the `_fs_to_record_path` function. When
we build the dictionary of installed files, we call it with a base
directory, while when build the set of modified files, we call it without a
base directory. As a result, the values of `installed` do not match the
elements of `changed`, and `get_csv_rows_for_installed` fails to identify
the rows that should be updated.
Fix this by ensuring that `_fs_to_record_path` is always called with a base
directory. `_record_to_fs_path` also needs a a base directory parameter to
be able to transform the path back into an absolute path, so add one.
Given an input tag list of size `n` and wheel file tags of size `m` this
method is currently `O(n*m)` because it iterates over the set of file tags
then for each file tag it iterates over the input tags.
We can do much better and get `O(m)` time complexity by iterating the input
tags instead, and doing a cheap `O(1)` membership test among the set of file
tags. As a side benefit, this also allows early-termination of the loop.
The impact of this seemingly trivial change is surprisingly big: for a run
of `pip-compile` on macOS, which calls this method many times with large
inputs this changes gives a ~50% speedup on end-to-end `pip-compile` time,
from ~8s down to ~4s!
This adds a --use-feature=truststore flag that, when specified on Python
3.10+ with truststore installed, switches pip to use truststore to
provide HTTPS certificate validation, instead of certifi. This allows
pip to verify certificates against custom certificates in the system
store.
truststore is deliberately NOT vendored because it is expected the
library to be under active development in the short term, and this
prevents users having to wait for a pip release to get potentially vital
bug fixes needed to be made in truststore.
Supplying the use-feature flag without installing truststore beforehand,
or on Python versions prior to 3.10, results in a command error.
This fallback is only triggered if the project has a `setup.py` file.
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Co-authored-by: Pradyun Gedam <pradyunsg@gmail.com>
Instead of a flag, make the option take an argument like this:
--root-user-action=ignore
This allows us to add more alternatives in the future, for example to
emit a hard error when a root user is detected.
Also re-label the news fragment to point to the issue instead of the PR
that introduced the option.
These were intended to help users transition when the default behaviour
changed to no longer perform out-of-tree builds. The transition is now
considered complete.