We explicitly propagate --use-feature options from req files upwards.
This is not strictly necessary for the option to be enabled, because
of the default value is a global list, but that implicit behaviour is
certainly accidental, so we make it explicit, with a test.
This patch adds support for `--use-feature` in requirements files
so that a project that wants all contributors using the same pip
features can specify it in the requirements file. For example, to ensure
a requirements file uses the new resolver:
```
--use-feature=2020-resolver
boto3
boto3==1.13.13
```
This is a new version of #8293.
Toward minimizing style changes in the overall PR diff, or toward
consistency with the future use of black (in cases where I wasn't sure
of a good way to minimize the diff).
This is a much better location for these errors, since they're in a much
more visible spot. We've had reports in the past of users missing these
messages, and changing where we present these warnings should help
resolve that issue.
We do lose the ability for an advanced user to potentially see the
warning and abort installation before the conflicts are introduced, but
given that we don't even pause for input, I don't think that's a strong
argument and neither do I view this as necessary.
The duplication of this code isn't really that bad, but saying
"pip check" makes it ambigous which file is relevant. Changing to
reference the exact filename makes this clearer.
This requirements format does not conform to PEP-508. Currently the
extras specified like this work by accident (because _strip_extras()
also parses them). The version checks end up being done with a
misparsed version '1.0[extra]' -- this is not changed in this commit.
Add deprecation warning and fix the corresponding resolver test. Add a
command line test.
Note that we really only check that the Requirement has SpecifierSet
with a specifier that ends in a ']'. A valid version number cannot
contain ']' and no wheels currently on pypi have versions ending in ']'.
Moving this value up from `_install_wheel` means that we do not need to
pass `req_description` anymore. This will also let us move our
entrypoint error handling around without worrying about losing the
context from the previous message.
Now we rely solely on the list of RECORD-like paths derived from the
filesystem, and can easily trade out the implementation for one that
comes from the wheel file directly.
At the beginning of our wheel processing we are going to have the list
of contained files. By splitting this into its own function, and
deriving it from disk in the same way it will appear in the zip, we can
incrementally refactor our approach using the same interface that will
be available at that time.
We start with the root-scheme paths (that end up in lib_dir) first.
When we start processing files directly from the wheel, all we will have
are the files with their zip path (which should match a `RECORD`
entry). Separating this from the source file path (used for copying)
and annotating it with our `RecordPath` type makes it clear what the
format of this public property is, and that it should match what is in
`RECORD`.
We always pass a file path to this function, so assert as much. We want
the return type to be consistent so we can assign the result to
non-Optional types.
"getting files" is one of the places that requires files to be on disk.
By extracting this out of `clobber` we can make it simpler and then
trade it out for a zip-based implementation.
Hiding the file-specific implementation we currently use will let us
trade out the implementation for a zip-backed one later. We can also use
this interface to represent the other kinds of files that we have to
generate as part of wheel installation.
We use a Protocol instead of a base class because there's no need for
shared behavior right now, and using Protocol is less verbose.
By removing this dependency of the "file installation" part of `clobber`
on the "file finding" part of `clobber`, we can more easily factor out
the "file installation" part.
Dropping the top-level directory creation allows us to make the
processing completely dependent on files to be installed, and not on the
top-level directory they happen to be installed in.
We already create the parent directory in the loop below, so this call
should be redundant for files that get installed.
For some time we have not needed to pre-emptively unpack wheels as part
of metadata processing, but kept the existing logic because the
behavior would start to diverge more for different package types. In
this case, though, removing the special cases for wheels makes this
logic a bit simpler, so it is worth doing.
This check only applies to explicit requirements since we avoid
downloading the dist from finder altogether when there is a matching
installation (although the check wouldn’t change the behaviour in that
case anyway).
We can do this when we build the `ExplicitRequirement` instead, like how
we did for `SpecifierRequirement`, but that would require us to resolve
the direct requirement’s version eagerly, which I don’t want to.
The implemented approach checks the version only after resolution, at
which point the distribution is already built anyway and the operation
is cheap.
There are a few changes here:
1. The byte-compilation now occurs after we copy the root-scheme files
and files from any wheel data dirs
1. Instead of iterating over the files in the unpacked wheel directory,
we iterate over the installed files as they exist in the installation
path
2. In addition to asserting that pyc files were created, we also add
them to the list of installed files, so they will be included in RECORD
By compiling after installation, we no longer depend on a separate
temporary directory - this brings us closer to installing directly from
wheel files.
By compiling with source files as they exist in the installation output
directory, we no longer generate pyc files with an embedded randomized
temp directory - this means that wheel installs can be deterministic.
In order to add generated pyc files to the RECORD file for our package,
we need to know their path! To raise confidence that we're doing this
correctly, we assert the existence of the expected 'pyc' files while
still using the old installation logic.
Some valid reasons why pyc files may not be generated:
1. Syntax error in the installed Python files
2. There is already a pyc file in-place that isn't writable by the
current user
We don't fail installation in those cases today, and we wouldn't want to
change our behavior here, so we only assert that the pyc file was
created if `compileall.compile_file` indicates success.
`compileall.compile_file` returns a success parameter, but can return
"successful" without actually generating a pyc file if the input file
was filtered out and compilation was not attempted.
In our file processing we mirror that logic, to ensure that a truthy
success returned by `compileall.compile_file` actually indicates a file
was written.
We want to move towards having more control over the generation of pyc
files, which will allow us to provide deterministic installs and
generate pyc files without relying on an already-extracted wheel.
To that end, here we are stripping away one layer of abstraction,
`compileall.compile_dir`. `compileall.compile_dir` essentially recurses
through the provided directories and passes the files and args verbatim
to `compileall.compile_file`, so removing that layer means that we
directly call `compileall.compile_file`.
We make the assumption that we can successfully walk over the
source file tree, since we just wrote it, and omit the per-directory
traversal error handling done by `compileall.compile_dir`.
Since the Distribution pulls its data directly from the Wheel file,
without extracting intermediate files to disk, this brings us closer to
installing from Wheels without extracting everything.
This big chunk of code was independent of the rest of our wheel
installation process. Moving it out enforces that there are no
dependencies between it and the original function, and makes it easier
to read the original function.
This makes get_csv_rows_for_installed simpler, because it is not
modifying its arguments. We can also more easily refactor RECORD file
reading since it is now decoupled from getting the installed RECORD file
rows.
Reducing the scope of variables reduces possible dependencies between
parts of this function, and will make it easier to extract this section
into its own function.
This reduces our dependence on the files being extracted to the
filesystem.
Compare the name extraction to the similar code in
`utils.wheel.wheel_dist_info_dir`.
We don't need to give `.data` directories the same strict
treatment (yet) because it isn't inconvenient if there happen
to be multiple of them in a single Wheel file.
Currently we do processing in `get_entrypoints` so incoming text is more compatible
with `pkg_resources`. It turns out that `pkg_resources` is already doing the same normalization,
so we can omit it.
This simplifies `get_entrypoints`, opening the way for us to pass it a plain string instead
of a file path.
These comments are relevant to this function, since it is long
overdue for refactoring. This code isn't special in that regard, and we
should feel free to consider any piece of code eligible to be broken up
or put into a class. So we remove these comments in fairness to the rest
of the code, and to remove a distraction during upcoming code reviews.
Pradyun Gedam
Pradyun Gedam
Nguyễn Gia Phong
Tzu-ping Chung
Stéphane Bidoul
Pradyun Gedam
Oliver Mannion
Andy Freeland
Chris Hunt
Jussi Kukkonen
Devesh Kumar Singh
gutsytechster
Stéphane Bidoul
Xavier Fernandez