The modern virtual environment structure does not allow us to enable
"fake user site" while disabling the global site, so we need to do more
fine-grained configuration to correctly set up test environments for
each test case.
With this done, we can also properly support the stdlib venv ad the test
environment backend, since it basically works identically with modern
virtualenv. The incompatible_with_test_venv is thus removed.
The tests still don't run without distutils
because they require virtualenv < 20 (and virtualenv 16 uses distutils),
but at least they don't import distutils directly now.
Fixes https://github.com/pypa/pip/issues/11521
The code to do this already exists in `get_csv_rows_for_installed`, but it's
broken due to inconsistent usage of the `_fs_to_record_path` function. When
we build the dictionary of installed files, we call it with a base
directory, while when build the set of modified files, we call it without a
base directory. As a result, the values of `installed` do not match the
elements of `changed`, and `get_csv_rows_for_installed` fails to identify
the rows that should be updated.
Fix this by ensuring that `_fs_to_record_path` is always called with a base
directory. `_record_to_fs_path` also needs a a base directory parameter to
be able to transform the path back into an absolute path, so add one.
The pip-specific Path implementation has been removed, and all its
usages replaced by pathlib.Path. The tmpdir and tmpdir_factory fixtures
are also removed, and all usages are replaced by tmp_path and
tmp_path_factory, which use pathlib.Path.
The pip() function now also accepts pathlib.Path so we don't need to put
str() everywhere. Path arguments are coerced with os.fspath() into str.
Note that the functional test does not actually detect the behavioral
change of throwing unhandled `BadZipFile` → throwing unhandled
`InvalidWheel`, whereas the unit test does.
Use pyupgrade to convert simple string formatting to use f-string
syntax. pyupgrade is intentionally timid and will not create an f-string
if it would make the expression longer or if the substitution parameters
are anything but simple names or dotted names.
Originally we would throw an `AttributeError` if a bad scheme key was
used. After refactoring we would throw a `KeyError`, which isn't much
better. Now we call out the wheel being processed, scheme key we didn't
recognize, and provide a list of the valid scheme keys. This would
likely be useful for people developing/testing the wheel.
Previously our wheel installation process allowed wheels which contained
non-conforming contents in a contained .data directory.
After the refactoring to enable direct-from-wheel installation, pip
throws an exception when encountering these wheels, but does not include
any helpful information to pinpoint the cause.
Now if we encounter such a wheel, we trace an error that includes the
name of the requirement we're trying to install, the path to the wheel
file, the path we didn't understand, and a hint about what we expect.
Since this is the special part of this test. This gives us more
confidence that we're doing the right thing when removing the standalone
wheel file next.
This mainly deals with correctly recording the wheel content in the
RECORD metadata. This metadata file must be written in UTF-8, but the
actual files need to be installed to the filesystem, the encoding of
which is not (always) UTF-8. So we need to carefully handle file name
encoding/decoding when comparing RECORD entries to the actual file.
The fix here makes sure we always use the correct encoding by adding
strict type hints. The entries in RECORD is decoded/encoded with UTF-8
on the read/write boundaries to make sure we always deal with text
types. A type-hint-only type RecordPath is introduced to make sure this
is enforced (because Python 2 "helpfully" coerces str to unicode with
the wrong encoding).
We now extract all metadata files from the wheel directly into memory
and make them available to the wrapping pkg_resources.Distribution via
the DictMetadata introduced earlier.
shared_data avoids copying the entire data directory, so use it in cases
where we know pip won't have any opportunity to edit the data files
(where we're passing tmpdir for --find-links).
Any test using --find-links data.packages is potentially using several
packages. By copying specifically the packages we need, we can more
easily see the packages that each test depends on and avoid using the
`data` fixture.
* Edit subdirs of top-level instead of checking in each directory
Previously, we were checking whether the top of the relative path ended
with .data. Now, we do not recurse into those directories, so there's no
need to check every time.
* Store info_dir in separate variable
Instead of working with a list everywhere, we use the single info_dir.
* Separate variables for info_dir and the destination path
* Use destination .dist-info dir only when needed
By initially storing just the name of the folder we ensure our code is
agnostic to the destination, so it'll be easier to install from a zip
later.
* Use os.listdir instead of os.walk for wheel dir population
Since we only execute any code when basedir == '', we only need the
top-level directories.
* Inline data_dirs calculation
* Inline info_dirs calculation
Previously we were restricting to a single .dist-info directory anywhere
in the unpacked wheel directory. That was incorrect since only a
top-level .dist-info directory indicates a contained "package". Now we
limit our restriction to top-level .dist-info directories.
This aligns with the previous behavior that would have enforced the
found .dist-info directory starting with the name of the package.
We raise UnsupportedWheel because it looks better in output than the
AssertionError (which includes traceback).
Our isolation logic for venv isn't correct and that is causing these
tests to fail. The culprits for this are:
tests/lib/venv.py::VirtualEnvironment.user_site_packages
tests/lib/venv.py::VirtualEnvironment.sitecustomize
Both these together are supposed to create an environment to isolate the
tests. However, they were written for virtualenv and make assumptions
that are not true for environments created with venv. Until we can fix
VirtualEnvironment to properly isolate the test from the underlying test
environment when using venv, these tests will continue to fail.
This is blocking an important bugfix for users facing issues with since
pip is installing packages into `--user` when run in a venv, even when
`--user` isn't visible from that environment.
As a temporary band-aid for this problem, I'm skipping these tests to
unblock us from shipping the bugfix for the aforementioned issue.
The test isolation logic should be fixed to work for venv. Once such a
fix is made, this commit should be reverted.
Damian Shaw
Stéphane Bidoul
Tzu-ping Chung
Miro Hrončok
Роман Донченко
q0w
Jon Dufresne
Lukas Juhrich
Pradyun Gedam
Hugo van Kemenade
Chris Hunt
gutsytechster
Paul Moore
Devesh Kumar Singh
Pradyun Gedam
Chris Jerdonek
Albert Tugushev