The object returned by NamedTemporaryFile delegates all functions calls
to the underlying file so can avoid the type override by relying on IO
methods.
Use pyupgrade to convert simple string formatting to use f-string
syntax. pyupgrade is intentionally timid and will not create an f-string
if it would make the expression longer or if the substitution parameters
are anything but simple names or dotted names.
Originally we would throw an `AttributeError` if a bad scheme key was
used. After refactoring we would throw a `KeyError`, which isn't much
better. Now we call out the wheel being processed, scheme key we didn't
recognize, and provide a list of the valid scheme keys. This would
likely be useful for people developing/testing the wheel.
Previously our wheel installation process allowed wheels which contained
non-conforming contents in a contained .data directory.
After the refactoring to enable direct-from-wheel installation, pip
throws an exception when encountering these wheels, but does not include
any helpful information to pinpoint the cause.
Now if we encounter such a wheel, we trace an error that includes the
name of the requirement we're trying to install, the path to the wheel
file, the path we didn't understand, and a hint about what we expect.
Moving this value up from `_install_wheel` means that we do not need to
pass `req_description` anymore. This will also let us move our
entrypoint error handling around without worrying about losing the
context from the previous message.
Now we rely solely on the list of RECORD-like paths derived from the
filesystem, and can easily trade out the implementation for one that
comes from the wheel file directly.
At the beginning of our wheel processing we are going to have the list
of contained files. By splitting this into its own function, and
deriving it from disk in the same way it will appear in the zip, we can
incrementally refactor our approach using the same interface that will
be available at that time.
We start with the root-scheme paths (that end up in lib_dir) first.
When we start processing files directly from the wheel, all we will have
are the files with their zip path (which should match a `RECORD`
entry). Separating this from the source file path (used for copying)
and annotating it with our `RecordPath` type makes it clear what the
format of this public property is, and that it should match what is in
`RECORD`.
We always pass a file path to this function, so assert as much. We want
the return type to be consistent so we can assign the result to
non-Optional types.
"getting files" is one of the places that requires files to be on disk.
By extracting this out of `clobber` we can make it simpler and then
trade it out for a zip-based implementation.
Hiding the file-specific implementation we currently use will let us
trade out the implementation for a zip-backed one later. We can also use
this interface to represent the other kinds of files that we have to
generate as part of wheel installation.
We use a Protocol instead of a base class because there's no need for
shared behavior right now, and using Protocol is less verbose.
By removing this dependency of the "file installation" part of `clobber`
on the "file finding" part of `clobber`, we can more easily factor out
the "file installation" part.
Dropping the top-level directory creation allows us to make the
processing completely dependent on files to be installed, and not on the
top-level directory they happen to be installed in.
We already create the parent directory in the loop below, so this call
should be redundant for files that get installed.
There are a few changes here:
1. The byte-compilation now occurs after we copy the root-scheme files
and files from any wheel data dirs
1. Instead of iterating over the files in the unpacked wheel directory,
we iterate over the installed files as they exist in the installation
path
2. In addition to asserting that pyc files were created, we also add
them to the list of installed files, so they will be included in RECORD
By compiling after installation, we no longer depend on a separate
temporary directory - this brings us closer to installing directly from
wheel files.
By compiling with source files as they exist in the installation output
directory, we no longer generate pyc files with an embedded randomized
temp directory - this means that wheel installs can be deterministic.
In order to add generated pyc files to the RECORD file for our package,
we need to know their path! To raise confidence that we're doing this
correctly, we assert the existence of the expected 'pyc' files while
still using the old installation logic.
Some valid reasons why pyc files may not be generated:
1. Syntax error in the installed Python files
2. There is already a pyc file in-place that isn't writable by the
current user
We don't fail installation in those cases today, and we wouldn't want to
change our behavior here, so we only assert that the pyc file was
created if `compileall.compile_file` indicates success.
`compileall.compile_file` returns a success parameter, but can return
"successful" without actually generating a pyc file if the input file
was filtered out and compilation was not attempted.
In our file processing we mirror that logic, to ensure that a truthy
success returned by `compileall.compile_file` actually indicates a file
was written.
We want to move towards having more control over the generation of pyc
files, which will allow us to provide deterministic installs and
generate pyc files without relying on an already-extracted wheel.
To that end, here we are stripping away one layer of abstraction,
`compileall.compile_dir`. `compileall.compile_dir` essentially recurses
through the provided directories and passes the files and args verbatim
to `compileall.compile_file`, so removing that layer means that we
directly call `compileall.compile_file`.
We make the assumption that we can successfully walk over the
source file tree, since we just wrote it, and omit the per-directory
traversal error handling done by `compileall.compile_dir`.
Since the Distribution pulls its data directly from the Wheel file,
without extracting intermediate files to disk, this brings us closer to
installing from Wheels without extracting everything.
This big chunk of code was independent of the rest of our wheel
installation process. Moving it out enforces that there are no
dependencies between it and the original function, and makes it easier
to read the original function.
Jon Dufresne
Pradyun Gedam
Hugo van Kemenade
Pradyun Gedam
Tzu-ping Chung
Chris Hunt
Nguyễn Gia Phong