Instead of an early return, we fall through to the existing check at the
end of this function. This aligns our treatment of `_fetch_metadata` and
`_prepare_distribution`.
Since `_prepare` is called in two places, we preserve the
`if self._dist is not None` protection above the new call to
`_fetch_metadata`. The second `if` in `_prepare` handles the early
return required when processing a lazy wheel.
Now that `_dist` is only set on success, we can use it to guard against
repeated execution instead of `_prepared`. As a result there are now only
two possible outcomes for calling `dist`:
1. `_dist` set and returned - lazy and non-lazy req
2. `_dist` not set and exception raised - bad lazy or bad non-lazy req
Previously a call to `_fetch_metadata` could result in several possible
outcomes:
1. `_dist` set, `_provided` not set, dist returned - for lazy wheels
2. `_dist` set, `_provided` not set, exception - for bad lazy wheels
3. `_dist` not set, `_provided` not set, exception - for non-lazy req
exceptions
4. `_dist` set, `_provided` not set, exception - for bad non-lazy reqs
5. `_dist` set, `_provided` set, dist returned - for non-lazy reqs
and probably more.
Our intent is to use `_dist` being set as the indicator of "this
requirement has been fully processed successfully" and discard
`_prepared`, since we don't actually rely on any of the other states
(they simply lead to a failure or in the future a retry).
This makes it more consistent with how error "summary" lines look.
eg:
IndexError: list index out of range
ModuleNotFoundError: No module named 'notamodule'
SVN has multiple distributions on Windows, e.g. SlikSVN, CollabNet. Some
of them suffix the version with a "-{distro}" part, which causes the
previous implementation to fail.
This patch removes that final part and make the version logic work.
The previous implementation uses pkg_resources.get_distribution(), which
does not canonicalize the package name correctly, and fails when
combined with pip's own get_distribution(), which does canonicalize
names. This makes InstallRequirement.check_if_exists() only use pip's
own canonicalization logic so different package name forms are matched
as expected.
Originally we would throw an `AttributeError` if a bad scheme key was
used. After refactoring we would throw a `KeyError`, which isn't much
better. Now we call out the wheel being processed, scheme key we didn't
recognize, and provide a list of the valid scheme keys. This would
likely be useful for people developing/testing the wheel.
Previously our wheel installation process allowed wheels which contained
non-conforming contents in a contained .data directory.
After the refactoring to enable direct-from-wheel installation, pip
throws an exception when encountering these wheels, but does not include
any helpful information to pinpoint the cause.
Now if we encounter such a wheel, we trace an error that includes the
name of the requirement we're trying to install, the path to the wheel
file, the path we didn't understand, and a hint about what we expect.
We explicitly propagate --use-feature options from req files upwards.
This is not strictly necessary for the option to be enabled, because
of the default value is a global list, but that implicit behaviour is
certainly accidental, so we make it explicit, with a test.
This is a fix for the sole failing test in the CI for these changes (in
tests/functional/test_debug.py::test_debug__library_versions). The
failure took me a fair bit of time to diagnose, but it looks like the
issue is that we're strictly comparing versions as strings. This is a
bad idea when they're not normalized.
This patch adds support for `--use-feature` in requirements files
so that a project that wants all contributors using the same pip
features can specify it in the requirements file. For example, to ensure
a requirements file uses the new resolver:
```
--use-feature=2020-resolver
boto3
boto3==1.13.13
```
This is a new version of #8293.
Toward minimizing style changes in the overall PR diff, or toward
consistency with the future use of black (in cases where I wasn't sure
of a good way to minimize the diff).
This is a much better location for these errors, since they're in a much
more visible spot. We've had reports in the past of users missing these
messages, and changing where we present these warnings should help
resolve that issue.
We do lose the ability for an advanced user to potentially see the
warning and abort installation before the conflicts are introduced, but
given that we don't even pause for input, I don't think that's a strong
argument and neither do I view this as necessary.
The duplication of this code isn't really that bad, but saying
"pip check" makes it ambigous which file is relevant. Changing to
reference the exact filename makes this clearer.
This requirements format does not conform to PEP-508. Currently the
extras specified like this work by accident (because _strip_extras()
also parses them). The version checks end up being done with a
misparsed version '1.0[extra]' -- this is not changed in this commit.
Add deprecation warning and fix the corresponding resolver test. Add a
command line test.
Note that we really only check that the Requirement has SpecifierSet
with a specifier that ends in a ']'. A valid version number cannot
contain ']' and no wheels currently on pypi have versions ending in ']'.
Moving this value up from `_install_wheel` means that we do not need to
pass `req_description` anymore. This will also let us move our
entrypoint error handling around without worrying about losing the
context from the previous message.
Now we rely solely on the list of RECORD-like paths derived from the
filesystem, and can easily trade out the implementation for one that
comes from the wheel file directly.
At the beginning of our wheel processing we are going to have the list
of contained files. By splitting this into its own function, and
deriving it from disk in the same way it will appear in the zip, we can
incrementally refactor our approach using the same interface that will
be available at that time.
We start with the root-scheme paths (that end up in lib_dir) first.
When we start processing files directly from the wheel, all we will have
are the files with their zip path (which should match a `RECORD`
entry). Separating this from the source file path (used for copying)
and annotating it with our `RecordPath` type makes it clear what the
format of this public property is, and that it should match what is in
`RECORD`.
We always pass a file path to this function, so assert as much. We want
the return type to be consistent so we can assign the result to
non-Optional types.
"getting files" is one of the places that requires files to be on disk.
By extracting this out of `clobber` we can make it simpler and then
trade it out for a zip-based implementation.
Hiding the file-specific implementation we currently use will let us
trade out the implementation for a zip-backed one later. We can also use
this interface to represent the other kinds of files that we have to
generate as part of wheel installation.
We use a Protocol instead of a base class because there's no need for
shared behavior right now, and using Protocol is less verbose.
By removing this dependency of the "file installation" part of `clobber`
on the "file finding" part of `clobber`, we can more easily factor out
the "file installation" part.
Dropping the top-level directory creation allows us to make the
processing completely dependent on files to be installed, and not on the
top-level directory they happen to be installed in.
We already create the parent directory in the loop below, so this call
should be redundant for files that get installed.
For some time we have not needed to pre-emptively unpack wheels as part
of metadata processing, but kept the existing logic because the
behavior would start to diverge more for different package types. In
this case, though, removing the special cases for wheels makes this
logic a bit simpler, so it is worth doing.
This check only applies to explicit requirements since we avoid
downloading the dist from finder altogether when there is a matching
installation (although the check wouldn’t change the behaviour in that
case anyway).
We can do this when we build the `ExplicitRequirement` instead, like how
we did for `SpecifierRequirement`, but that would require us to resolve
the direct requirement’s version eagerly, which I don’t want to.
The implemented approach checks the version only after resolution, at
which point the distribution is already built anyway and the operation
is cheap.
There are a few changes here:
1. The byte-compilation now occurs after we copy the root-scheme files
and files from any wheel data dirs
1. Instead of iterating over the files in the unpacked wheel directory,
we iterate over the installed files as they exist in the installation
path
2. In addition to asserting that pyc files were created, we also add
them to the list of installed files, so they will be included in RECORD
By compiling after installation, we no longer depend on a separate
temporary directory - this brings us closer to installing directly from
wheel files.
By compiling with source files as they exist in the installation output
directory, we no longer generate pyc files with an embedded randomized
temp directory - this means that wheel installs can be deterministic.
In order to add generated pyc files to the RECORD file for our package,
we need to know their path! To raise confidence that we're doing this
correctly, we assert the existence of the expected 'pyc' files while
still using the old installation logic.
Some valid reasons why pyc files may not be generated:
1. Syntax error in the installed Python files
2. There is already a pyc file in-place that isn't writable by the
current user
We don't fail installation in those cases today, and we wouldn't want to
change our behavior here, so we only assert that the pyc file was
created if `compileall.compile_file` indicates success.
`compileall.compile_file` returns a success parameter, but can return
"successful" without actually generating a pyc file if the input file
was filtered out and compilation was not attempted.
In our file processing we mirror that logic, to ensure that a truthy
success returned by `compileall.compile_file` actually indicates a file
was written.
We want to move towards having more control over the generation of pyc
files, which will allow us to provide deterministic installs and
generate pyc files without relying on an already-extracted wheel.
To that end, here we are stripping away one layer of abstraction,
`compileall.compile_dir`. `compileall.compile_dir` essentially recurses
through the provided directories and passes the files and args verbatim
to `compileall.compile_file`, so removing that layer means that we
directly call `compileall.compile_file`.
We make the assumption that we can successfully walk over the
source file tree, since we just wrote it, and omit the per-directory
traversal error handling done by `compileall.compile_dir`.
Since the Distribution pulls its data directly from the Wheel file,
without extracting intermediate files to disk, this brings us closer to
installing from Wheels without extracting everything.
