McSinyx/pip
1
1
Fork 0
mirror of https://github.com/pypa/pip synced 2023-12-13 21:30:23 +01:00
Code Issues Releases Activity

Merge pull request #5743 from pfmoore/pep517

Browse source 
PEP 517 implementation
This commit is contained in:
Paul Moore 2018-11-15 09:32:17 +00:00 committed by GitHub
parent 507b90e117 f06a0cb560
commit 3a77bd667c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
36 changed files with 869 additions and 235 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

105
docs/html/reference/pip.rst
View file

@ -71,13 +71,13 @@ when decision is needed.
Build System Interface
======================
Pip builds packages by invoking the build system. Presently, the only supported
build system is ``setuptools``, but in the future, pip will support :pep:`517`
which allows projects to specify an alternative build system in a
``pyproject.toml`` file. As well as package building, the build system is also
invoked to install packages direct from source. This is handled by invoking
the build system to build a wheel, and then installing from that wheel. The
built wheel is cached locally by pip to avoid repeated identical builds.
Pip builds packages by invoking the build system. By default, builds will use
``setuptools``, but if a project specifies a different build system using a
``pyproject.toml`` file, as per :pep:`517`, pip will use that instead. As well
as package building, the build system is also invoked to install packages
direct from source. This is handled by invoking the build system to build a
wheel, and then installing from that wheel. The built wheel is cached locally
by pip to avoid repeated identical builds.
The current interface to the build system is via the ``setup.py`` command line
script - all build actions are defined in terms of the specific ``setup.py``
@ -86,13 +86,16 @@ command line that will be run to invoke the required action.
Setuptools Injection
~~~~~~~~~~~~~~~~~~~~
As noted above, the supported build system is ``setuptools``. However, not all
packages use ``setuptools`` in their build scripts. To support projects that
use "pure ``distutils``", pip injects ``setuptools`` into ``sys.modules``
before invoking ``setup.py``. The injection should be transparent to
``distutils``-based projects, but 3rd party build tools wishing to provide a
``setup.py`` emulating the commands pip requires may need to be aware that it
takes place.
When :pep:`517` is not used, the supported build system is ``setuptools``.
However, not all packages use ``setuptools`` in their build scripts. To support
projects that use "pure ``distutils``", pip injects ``setuptools`` into
``sys.modules`` before invoking ``setup.py``. The injection should be
transparent to ``distutils``-based projects, but 3rd party build tools wishing
to provide a ``setup.py`` emulating the commands pip requires may need to be
aware that it takes place.
Projects using :pep:`517` *must* explicitly use setuptools - pip does not do
the above injection process in this case.
Build System Output
~~~~~~~~~~~~~~~~~~~
@ -113,13 +116,20 @@ unexpected byte sequences to Python-style hexadecimal escape sequences
(``"\x80\xff"``, etc). However, it is still possible for output to be displayed
using an incorrect encoding (mojibake).
PEP 518 Support
~~~~~~~~~~~~~~~
Under :pep:`517`, handling of build tool output is the backend's responsibility,
and pip simply displays the output produced by the backend. (Backends, however,
will likely still have to address the issues described above).
As of 10.0, pip supports projects declaring dependencies that are required at
install time using a ``pyproject.toml`` file, in the form described in
:pep:`518`. When building a project, pip will install the required dependencies
locally, and make them available to the build process.
PEP 517 and 518 Support
~~~~~~~~~~~~~~~~~~~~~~~
As of version 10.0, pip supports projects declaring dependencies that are
required at install time using a ``pyproject.toml`` file, in the form described
in :pep:`518`. When building a project, pip will install the required
dependencies locally, and make them available to the build process.
Furthermore, from version 19.0 onwards, pip supports projects specifying the
build backend they use in ``pyproject.toml``, in the form described in
:pep:`517`.
When making build requirements available, pip does so in an *isolated
environment*. That is, pip does not install those requirements into the user's
@ -137,24 +147,49 @@ can be problematic. If this is the case, pip provides a
flag are responsible for ensuring the build environment is managed
appropriately.
By default, pip will continue to use the legacy (``setuptools`` based) build
processing for projects that do not have a ``pyproject.toml`` file. Projects
with a ``pyproject.toml`` file will use a :pep:`517` backend. Projects with
a ``pyproject.toml`` file, but which don't have a ``build-system`` section,
will be assumed to have the following backend settings::
[build-system]
requires = ["setuptools>=40.2.0", "wheel"]
build-backend = "setuptools.build_meta"
.. note::
``setuptools`` 40.2.0 is the first version of setuptools with full
:pep:`517` support.
If a project has ``[build-system]``, but no ``build-backend``, pip will use
``setuptools.build_meta``, but will assume the project requirements include
``setuptools>=40.2.0`` and ``wheel`` (and will report an error if not).
If a user wants to explicitly request :pep:`517` handling even though a project
doesn't have a ``pyproject.toml`` file, this can be done using the
``--use-pep517`` command line option. Similarly, to request legacy processing
even though ``pyproject.toml`` is present, the ``--no-use-pep517`` option is
available (although obviously it is an error to choose ``--no-use-pep517`` if
the project has no ``setup.py``, or explicitly requests a build backend). As
with other command line flags, pip recognises the ``PIP_USE_PEP517``
environment veriable and a ``use-pep517`` config file option (set to true or
false) to set this option globally. Note that overriding pip's choice of
whether to use :pep:`517` processing in this way does *not* affect whether pip
will use an isolated build environment (which is controlled via
``--no-build-isolation`` as noted above).
Except in the case noted above (projects with no :pep:`518` ``[build-system]``
section in ``pyproject.toml``), pip will never implicitly install a build
system. Projects **must** ensure that the correct build system is listed in
their ``requires`` list (this applies even if pip assumes that the
``setuptools`` backend is being used, as noted above).
.. _pep-518-limitations:
**Limitations**:
**Historical Limitations**:
* until :pep:`517` support is added, ``setuptools`` and ``wheel`` **must** be
included in the list of build requirements: pip will assume these as default,
but will not automatically add them to the list of build requirements if
explicitly defined in ``pyproject.toml``.
* the current implementation only support installing build requirements from
wheels: this is a technical limitation of the implementation - source
installs would require a build step of their own, potentially recursively
triggering another :pep:`518` dependency installation process. The possible
unbounded recursion involved was not considered acceptable, and so
installation of build dependencies from source has been disabled until a safe
resolution of this issue is found.
* ``pip<18.0``: only support installing build requirements from wheels, and
* ``pip<18.0``: only supports installing build requirements from wheels, and
does not support the use of environment markers and extras (only version
specifiers are respected).

1
news/5743.feature Normal file
View file

@ -0,0 +1 @@
Implement PEP 517 (allow projects to specify a build backend via pyproject.toml).

165
src/pip/_internal/build_env.py
View file

@ -5,6 +5,7 @@ import logging
import os
import sys
import textwrap
from collections import OrderedDict
from distutils.sysconfig import get_python_lib
from sysconfig import get_paths
@ -18,6 +19,25 @@ from pip._internal.utils.ui import open_spinner
logger = logging.getLogger(__name__)
class _Prefix:
def __init__(self, path):
self.path = path
self.setup = False
self.bin_dir = get_paths(
'nt' if os.name == 'nt' else 'posix_prefix',
vars={'base': path, 'platbase': path}
)['scripts']
# Note: prefer distutils' sysconfig to get the
# library paths so PyPy is correctly supported.
purelib = get_python_lib(plat_specific=0, prefix=path)
platlib = get_python_lib(plat_specific=1, prefix=path)
if purelib == platlib:
self.lib_dirs = [purelib]
else:
self.lib_dirs = [purelib, platlib]
class BuildEnvironment(object):
"""Creates and manages an isolated environment to install build deps
"""
@ -26,86 +46,113 @@ class BuildEnvironment(object):
self._temp_dir = TempDirectory(kind="build-env")
self._temp_dir.create()
@property
def path(self):
return self._temp_dir.path
self._prefixes = OrderedDict((
(name, _Prefix(os.path.join(self._temp_dir.path, name)))
for name in ('normal', 'overlay')
))
def __enter__(self):
self.save_path = os.environ.get('PATH', None)
self.save_pythonpath = os.environ.get('PYTHONPATH', None)
self.save_nousersite = os.environ.get('PYTHONNOUSERSITE', None)
self._bin_dirs = []
self._lib_dirs = []
for prefix in reversed(list(self._prefixes.values())):
self._bin_dirs.append(prefix.bin_dir)
self._lib_dirs.extend(prefix.lib_dirs)
install_scheme = 'nt' if (os.name == 'nt') else 'posix_prefix'
install_dirs = get_paths(install_scheme, vars={
'base': self.path,
'platbase': self.path,
})
scripts = install_dirs['scripts']
if self.save_path:
os.environ['PATH'] = scripts + os.pathsep + self.save_path
else:
os.environ['PATH'] = scripts + os.pathsep + os.defpath
# Note: prefer distutils' sysconfig to get the
# library paths so PyPy is correctly supported.
purelib = get_python_lib(plat_specific=0, prefix=self.path)
platlib = get_python_lib(plat_specific=1, prefix=self.path)
if purelib == platlib:
lib_dirs = purelib
else:
lib_dirs = purelib + os.pathsep + platlib
if self.save_pythonpath:
os.environ['PYTHONPATH'] = lib_dirs + os.pathsep + \
self.save_pythonpath
else:
os.environ['PYTHONPATH'] = lib_dirs
os.environ['PYTHONNOUSERSITE'] = '1'
# Ensure .pth files are honored.
with open(os.path.join(purelib, 'sitecustomize.py'), 'w') as fp:
# Customize site to:
# - ensure .pth files are honored
# - prevent access to system site packages
system_sites = {
os.path.normcase(site) for site in (
get_python_lib(plat_specific=0),
get_python_lib(plat_specific=1),
)
}
self._site_dir = os.path.join(self._temp_dir.path, 'site')
if not os.path.exists(self._site_dir):
os.mkdir(self._site_dir)
with open(os.path.join(self._site_dir, 'sitecustomize.py'), 'w') as fp:
fp.write(textwrap.dedent(
'''
import site
site.addsitedir({!r})
'''
).format(purelib))
import os, site, sys
return self.path
# First, drop system-sites related paths.
original_sys_path = sys.path[:]
known_paths = set()
for path in {system_sites!r}:
site.addsitedir(path, known_paths=known_paths)
system_paths = set(
os.path.normcase(path)
for path in sys.path[len(original_sys_path):]
)
original_sys_path = [
path for path in original_sys_path
if os.path.normcase(path) not in system_paths
]
sys.path = original_sys_path
# Second, add lib directories.
# ensuring .pth file are processed.
for path in {lib_dirs!r}:
assert not path in sys.path
site.addsitedir(path)
'''
).format(system_sites=system_sites, lib_dirs=self._lib_dirs))
def __enter__(self):
self._save_env = {
name: os.environ.get(name, None)
for name in ('PATH', 'PYTHONNOUSERSITE', 'PYTHONPATH')
}
path = self._bin_dirs[:]
old_path = self._save_env['PATH']
if old_path:
path.extend(old_path.split(os.pathsep))
pythonpath = [self._site_dir]
os.environ.update({
'PATH': os.pathsep.join(path),
'PYTHONNOUSERSITE': '1',
'PYTHONPATH': os.pathsep.join(pythonpath),
})
def __exit__(self, exc_type, exc_val, exc_tb):
def restore_var(varname, old_value):
for varname, old_value in self._save_env.items():
if old_value is None:
os.environ.pop(varname, None)
else:
os.environ[varname] = old_value
restore_var('PATH', self.save_path)
restore_var('PYTHONPATH', self.save_pythonpath)
restore_var('PYTHONNOUSERSITE', self.save_nousersite)
def cleanup(self):
self._temp_dir.cleanup()
def missing_requirements(self, reqs):
"""Return a list of the requirements from reqs that are not present
def check_requirements(self, reqs):
"""Return 2 sets:
- conflicting requirements: set of (installed, wanted) reqs tuples
- missing requirements: set of reqs
"""
missing = []
with self:
ws = WorkingSet(os.environ["PYTHONPATH"].split(os.pathsep))
missing = set()
conflicting = set()
if reqs:
ws = WorkingSet(self._lib_dirs)
for req in reqs:
try:
if ws.find(Requirement.parse(req)) is None:
missing.append(req)
except VersionConflict:
missing.append(req)
return missing
missing.add(req)
except VersionConflict as e:
conflicting.add((str(e.args[0].as_requirement()),
str(e.args[1])))
return conflicting, missing
def install_requirements(self, finder, requirements, message):
def install_requirements(self, finder, requirements, prefix, message):
prefix = self._prefixes[prefix]
assert not prefix.setup
prefix.setup = True
if not requirements:
return
args = [
sys.executable, os.path.dirname(pip_location), 'install',
'--ignore-installed', '--no-user', '--prefix', self.path,
'--ignore-installed', '--no-user', '--prefix', prefix.path,
'--no-warn-script-location',
]
if logger.getEffectiveLevel() <= logging.DEBUG:
@ -150,5 +197,5 @@ class NoOpBuildEnvironment(BuildEnvironment):
def cleanup(self):
pass
def install_requirements(self, finder, requirements, message):
def install_requirements(self, finder, requirements, prefix, message):
raise NotImplementedError()

5
src/pip/_internal/cli/base_command.py
View file

@ -232,6 +232,7 @@ class RequirementCommand(Command):
for req in args:
req_to_add = install_req_from_line(
req, None, isolated=options.isolated_mode,
use_pep517=options.use_pep517,
wheel_cache=wheel_cache
)
req_to_add.is_direct = True
@ -241,6 +242,7 @@ class RequirementCommand(Command):
req_to_add = install_req_from_editable(
req,
isolated=options.isolated_mode,
use_pep517=options.use_pep517,
wheel_cache=wheel_cache
)
req_to_add.is_direct = True
@ -250,7 +252,8 @@ class RequirementCommand(Command):
for req_to_add in parse_requirements(
filename,
finder=finder, options=options, session=session,
wheel_cache=wheel_cache):
wheel_cache=wheel_cache,
use_pep517=options.use_pep517):
req_to_add.is_direct = True
requirement_set.add_requirement(req_to_add)
# If --require-hashes was a line in a requirements file, tell

19
src/pip/_internal/cli/cmdoptions.py
View file

@ -612,6 +612,25 @@ no_build_isolation = partial(
'if this option is used.'
) # type: partial[Option]
use_pep517 = partial(
Option,
'--use-pep517',
dest='use_pep517',
action='store_true',
default=None,
help='Use PEP 517 for building source distributions '
'(use --no-use-pep517 to force legacy behaviour).'
) # type: Any
no_use_pep517 = partial(
Option,
'--no-use-pep517',
dest='use_pep517',
action='store_false',
default=None,
help=SUPPRESS_HELP
) # type: Any
install_options = partial(
Option,
'--install-option',

2
src/pip/_internal/commands/download.py
View file

@ -58,6 +58,8 @@ class DownloadCommand(RequirementCommand):
cmd_opts.add_option(cmdoptions.require_hashes())
cmd_opts.add_option(cmdoptions.progress_bar())
cmd_opts.add_option(cmdoptions.no_build_isolation())
cmd_opts.add_option(cmdoptions.use_pep517())
cmd_opts.add_option(cmdoptions.no_use_pep517())
cmd_opts.add_option(
'-d', '--dest', '--destination-dir', '--destination-directory',

61
src/pip/_internal/commands/install.py
View file

@ -30,12 +30,6 @@ from pip._internal.utils.misc import (
from pip._internal.utils.temp_dir import TempDirectory
from pip._internal.wheel import WheelBuilder
try:
import wheel
except ImportError:
wheel = None
logger = logging.getLogger(__name__)
@ -158,6 +152,8 @@ class InstallCommand(RequirementCommand):
cmd_opts.add_option(cmdoptions.ignore_requires_python())
cmd_opts.add_option(cmdoptions.no_build_isolation())
cmd_opts.add_option(cmdoptions.use_pep517())
cmd_opts.add_option(cmdoptions.no_use_pep517())
cmd_opts.add_option(cmdoptions.install_options())
cmd_opts.add_option(cmdoptions.global_options())
@ -314,6 +310,7 @@ class InstallCommand(RequirementCommand):
ignore_requires_python=options.ignore_requires_python,
ignore_installed=options.ignore_installed,
isolated=options.isolated_mode,
use_pep517=options.use_pep517
)
resolver.resolve(requirement_set)
@ -321,21 +318,51 @@ class InstallCommand(RequirementCommand):
modifying_pip=requirement_set.has_requirement("pip")
)
# If caching is disabled or wheel is not installed don't
# try to build wheels.
if wheel and options.cache_dir:
# build wheels before install.
wb = WheelBuilder(
finder, preparer, wheel_cache,
build_options=[], global_options=[],
)
# Ignore the result: a failed wheel will be
# installed from the sdist/vcs whatever.
# Consider legacy and PEP517-using requirements separately
legacy_requirements = []
pep517_requirements = []
for req in requirement_set.requirements.values():
if req.use_pep517:
pep517_requirements.append(req)
else:
legacy_requirements.append(req)
# We don't build wheels for legacy requirements if we
# don't have wheel installed or we don't have a cache dir
try:
import wheel # noqa: F401
build_legacy = bool(options.cache_dir)
except ImportError:
build_legacy = False
wb = WheelBuilder(
finder, preparer, wheel_cache,
build_options=[], global_options=[],
)
# Always build PEP 517 requirements
build_failures = wb.build(
pep517_requirements,
session=session, autobuilding=True
)
if build_legacy:
# We don't care about failures building legacy
# requirements, as we'll fall through to a direct
# install for those.
wb.build(
requirement_set.requirements.values(),
legacy_requirements,
session=session, autobuilding=True
)
# If we're using PEP 517, we cannot do a direct install
# so we fail here.
if build_failures:
raise InstallationError(
"Could not build wheels for {} which use" +
" PEP 517 and cannot be installed directly".format(
", ".join(r.name for r in build_failures)))
to_install = resolver.get_installation_order(
requirement_set
)

7
src/pip/_internal/commands/wheel.py
View file

@ -67,6 +67,8 @@ class WheelCommand(RequirementCommand):
help="Extra arguments to be supplied to 'setup.py bdist_wheel'.",
)
cmd_opts.add_option(cmdoptions.no_build_isolation())
cmd_opts.add_option(cmdoptions.use_pep517())
cmd_opts.add_option(cmdoptions.no_use_pep517())
cmd_opts.add_option(cmdoptions.constraints())
cmd_opts.add_option(cmdoptions.editable())
cmd_opts.add_option(cmdoptions.requirements())
@ -157,6 +159,7 @@ class WheelCommand(RequirementCommand):
ignore_requires_python=options.ignore_requires_python,
ignore_installed=True,
isolated=options.isolated_mode,
use_pep517=options.use_pep517
)
resolver.resolve(requirement_set)
@ -167,10 +170,10 @@ class WheelCommand(RequirementCommand):
global_options=options.global_options or [],
no_clean=options.no_clean,
)
wheels_built_successfully = wb.build(
build_failures = wb.build(
requirement_set.requirements.values(), session=session,
)
if not wheels_built_successfully:
if len(build_failures) != 0:
raise CommandError(
"Failed to build one or more wheels"
)

39
src/pip/_internal/operations/prepare.py
View file

@ -100,30 +100,53 @@ class IsSDist(DistAbstraction):
self.req.load_pyproject_toml()
should_isolate = self.req.use_pep517 and build_isolation
def _raise_conflicts(conflicting_with, conflicting_reqs):
raise InstallationError(
"Some build dependencies for %s conflict with %s: %s." % (
self.req, conflicting_with, ', '.join(
'%s is incompatible with %s' % (installed, wanted)
for installed, wanted in sorted(conflicting))))
if should_isolate:
# Isolate in a BuildEnvironment and install the build-time
# requirements.
self.req.build_env = BuildEnvironment()
self.req.build_env.install_requirements(
finder, self.req.pyproject_requires,
finder, self.req.pyproject_requires, 'overlay',
"Installing build dependencies"
)
missing = []
if self.req.requirements_to_check:
check = self.req.requirements_to_check
missing = self.req.build_env.missing_requirements(check)
conflicting, missing = self.req.build_env.check_requirements(
self.req.requirements_to_check
)
if conflicting:
_raise_conflicts("PEP 517/518 supported requirements",
conflicting)
if missing:
logger.warning(
"Missing build requirements in pyproject.toml for %s.",
self.req,
)
logger.warning(
"The project does not specify a build backend, and pip "
"cannot fall back to setuptools without %s.",
"The project does not specify a build backend, and "
"pip cannot fall back to setuptools without %s.",
" and ".join(map(repr, sorted(missing)))
)
# Install any extra build dependencies that the backend requests.
# This must be done in a second pass, as the pyproject.toml
# dependencies must be installed before we can call the backend.
with self.req.build_env:
# We need to have the env active when calling the hook.
self.req.spin_message = "Getting requirements to build wheel"
reqs = self.req.pep517_backend.get_requires_for_build_wheel()
conflicting, missing = self.req.build_env.check_requirements(reqs)
if conflicting:
_raise_conflicts("the backend dependencies", conflicting)
self.req.build_env.install_requirements(
finder, missing, 'normal',
"Installing backend dependencies"
)
self.req.run_egg_info()
self.req.prepare_metadata()
self.req.assert_source_matches_version()

17
src/pip/_internal/pyproject.py
View file

@ -88,7 +88,7 @@ def load_pyproject_toml(use_pep517, pyproject_toml, setup_py, req_name):
# assume the setuptools backend, and require wheel and a version
# of setuptools that supports that backend.
build_system = {
"requires": ["setuptools>=38.2.5", "wheel"],
"requires": ["setuptools>=40.2.0", "wheel"],
"build-backend": "setuptools.build_meta",
}
@ -131,14 +131,13 @@ def load_pyproject_toml(use_pep517, pyproject_toml, setup_py, req_name):
# (which is neede by the backend) in their requirements. So we
# make a note to check that those requirements are present once
# we have set up the environment.
# TODO: Review this - it's quite a lot of work to check for a very
# specific case. The problem is, that case is potentially quite
# common - projects that adopted PEP 518 early for the ability to
# specify requirements to execute setup.py, but never considered
# needing to mention the build tools themselves. The original PEP
# 518 code had a similar check (but implemented in a different
# way).
# This is quite a lot of work to check for a very specific case. But
# the problem is, that case is potentially quite common - projects that
# adopted PEP 518 early for the ability to specify requirements to
# execute setup.py, but never considered needing to mention the build
# tools themselves. The original PEP 518 code had a similar check (but
# implemented in a different way).
backend = "setuptools.build_meta"
check = ["setuptools>=38.2.5", "wheel"]
check = ["setuptools>=40.2.0", "wheel"]
return (requires, backend, check)

17
src/pip/_internal/req/constructors.py
View file

@ -145,8 +145,8 @@ def deduce_helpful_msg(req):
def install_req_from_editable(
editable_req, comes_from=None, isolated=False, options=None,
wheel_cache=None, constraint=False
editable_req, comes_from=None, use_pep517=None, isolated=False,
options=None, wheel_cache=None, constraint=False
):
name, url, extras_override = parse_editable(editable_req)
if url.startswith('file:'):
@ -166,6 +166,7 @@ def install_req_from_editable(
editable=True,
link=Link(url),
constraint=constraint,
use_pep517=use_pep517,
isolated=isolated,
options=options if options else {},
wheel_cache=wheel_cache,
@ -174,8 +175,8 @@ def install_req_from_editable(
def install_req_from_line(
name, comes_from=None, isolated=False, options=None, wheel_cache=None,
constraint=False
name, comes_from=None, use_pep517=None, isolated=False, options=None,
wheel_cache=None, constraint=False
):
"""Creates an InstallRequirement from a name, which might be a
requirement, directory containing 'setup.py', filename, or URL.
@ -264,7 +265,7 @@ def install_req_from_line(
return InstallRequirement(
req, comes_from, link=link, markers=markers,
isolated=isolated,
use_pep517=use_pep517, isolated=isolated,
options=options if options else {},
wheel_cache=wheel_cache,
constraint=constraint,
@ -273,7 +274,8 @@ def install_req_from_line(
def install_req_from_req(
req, comes_from=None, isolated=False, wheel_cache=None
req, comes_from=None, isolated=False, wheel_cache=None,
use_pep517=None
):
try:
req = Requirement(req)
@ -293,5 +295,6 @@ def install_req_from_req(
)
return InstallRequirement(
req, comes_from, isolated=isolated, wheel_cache=wheel_cache
req, comes_from, isolated=isolated, wheel_cache=wheel_cache,
use_pep517=use_pep517
)

10
src/pip/_internal/req/req_file.py
View file

@ -60,7 +60,8 @@ SUPPORTED_OPTIONS_REQ_DEST = [o().dest for o in SUPPORTED_OPTIONS_REQ]
def parse_requirements(filename, finder=None, comes_from=None, options=None,
session=None, constraint=False, wheel_cache=None):
session=None, constraint=False, wheel_cache=None,
use_pep517=None):
"""Parse a requirements file and yield InstallRequirement instances.
:param filename: Path or url of requirements file.
@ -71,6 +72,7 @@ def parse_requirements(filename, finder=None, comes_from=None, options=None,
:param constraint: If true, parsing a constraint file rather than
requirements file.
:param wheel_cache: Instance of pip.wheel.WheelCache
:param use_pep517: Value of the --use-pep517 option.
"""
if session is None:
raise TypeError(
@ -87,7 +89,7 @@ def parse_requirements(filename, finder=None, comes_from=None, options=None,
for line_number, line in lines_enum:
req_iter = process_line(line, filename, line_number, finder,
comes_from, options, session, wheel_cache,
constraint=constraint)
use_pep517=use_pep517, constraint=constraint)
for req in req_iter:
yield req
@ -108,7 +110,7 @@ def preprocess(content, options):
def process_line(line, filename, line_number, finder=None, comes_from=None,
options=None, session=None, wheel_cache=None,
constraint=False):
use_pep517=None, constraint=False):
"""Process a single requirements line; This can result in creating/yielding
requirements, or updating the finder.
@ -155,6 +157,7 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
req_options[dest] = opts.__dict__[dest]
yield install_req_from_line(
args_str, line_comes_from, constraint=constraint,
use_pep517=use_pep517,
isolated=isolated, options=req_options, wheel_cache=wheel_cache
)
@ -163,6 +166,7 @@ def process_line(line, filename, line_number, finder=None, comes_from=None,
isolated = options.isolated_mode if options else False
yield install_req_from_editable(
opts.editables[0], comes_from=line_comes_from,
use_pep517=use_pep517,
constraint=constraint, isolated=isolated, wheel_cache=wheel_cache
)

155
src/pip/_internal/req/req_install.py
View file

@ -50,7 +50,7 @@ class InstallRequirement(object):
"""
def __init__(self, req, comes_from, source_dir=None, editable=False,
link=None, update=True, markers=None,
link=None, update=True, markers=None, use_pep517=None,
isolated=False, options=None, wheel_cache=None,
constraint=False, extras=()):
assert req is None or isinstance(req, Requirement), req
@ -107,11 +107,16 @@ class InstallRequirement(object):
self.isolated = isolated
self.build_env = NoOpBuildEnvironment()
# For PEP 517, the directory where we request the project metadata
# gets stored. We need this to pass to build_wheel, so the backend
# can ensure that the wheel matches the metadata (see the PEP for
# details).
self.metadata_directory = None
# The static build requirements (from pyproject.toml)
self.pyproject_requires = None
# Build requirements that we will check are available
# TODO: We don't do this for --no-build-isolation. Should we?
self.requirements_to_check = []
# The PEP 517 backend we should use to build the project
@ -122,7 +127,7 @@ class InstallRequirement(object):
# and False. Before loading, None is valid (meaning "use the default").
# Setting an explicit value before loading pyproject.toml is supported,
# but after loading this flag should be treated as read only.
self.use_pep517 = None
self.use_pep517 = use_pep517
def __str__(self):
if self.req:
@ -311,6 +316,14 @@ class InstallRequirement(object):
self.source_dir = os.path.normpath(os.path.abspath(new_location))
self._egg_info_path = None
# Correct the metadata directory, if it exists
if self.metadata_directory:
old_meta = self.metadata_directory
rel = os.path.relpath(old_meta, start=old_location)
new_meta = os.path.join(new_location, rel)
new_meta = os.path.normpath(os.path.abspath(new_meta))
self.metadata_directory = new_meta
def remove_temporary_source(self):
"""Remove the source files from this requirement, if they are marked
for deletion"""
@ -437,40 +450,35 @@ class InstallRequirement(object):
self.pyproject_requires = requires
self.pep517_backend = Pep517HookCaller(self.setup_py_dir, backend)
def run_egg_info(self):
# Use a custom function to call subprocesses
self.spin_message = ""
def runner(cmd, cwd=None, extra_environ=None):
with open_spinner(self.spin_message) as spinner:
call_subprocess(
cmd,
cwd=cwd,
extra_environ=extra_environ,
show_stdout=False,
spinner=spinner
)
self.spin_message = ""
self.pep517_backend._subprocess_runner = runner
def prepare_metadata(self):
"""Ensure that project metadata is available.
Under PEP 517, call the backend hook to prepare the metadata.
Under legacy processing, call setup.py egg-info.
"""
assert self.source_dir
if self.name:
logger.debug(
'Running setup.py (path:%s) egg_info for package %s',
self.setup_py, self.name,
)
else:
logger.debug(
'Running setup.py (path:%s) egg_info for package from %s',
self.setup_py, self.link,
)
with indent_log():
script = SETUPTOOLS_SHIM % self.setup_py
base_cmd = [sys.executable, '-c', script]
if self.isolated:
base_cmd += ["--no-user-cfg"]
egg_info_cmd = base_cmd + ['egg_info']
# We can't put the .egg-info files at the root, because then the
# source code will be mistaken for an installed egg, causing
# problems
if self.editable:
egg_base_option = []
if self.use_pep517:
self.prepare_pep517_metadata()
else:
egg_info_dir = os.path.join(self.setup_py_dir, 'pip-egg-info')
ensure_dir(egg_info_dir)
egg_base_option = ['--egg-base', 'pip-egg-info']
with self.build_env:
call_subprocess(
egg_info_cmd + egg_base_option,
cwd=self.setup_py_dir,
show_stdout=False,
command_desc='python setup.py egg_info')
self.run_egg_info()
if not self.req:
if isinstance(parse_version(self.metadata["Version"]), Version):
@ -489,13 +497,66 @@ class InstallRequirement(object):
metadata_name = canonicalize_name(self.metadata["Name"])
if canonicalize_name(self.req.name) != metadata_name:
logger.warning(
'Running setup.py (path:%s) egg_info for package %s '
'Generating metadata for package %s '
'produced metadata for project name %s. Fix your '
'#egg=%s fragments.',
self.setup_py, self.name, metadata_name, self.name
self.name, metadata_name, self.name
)
self.req = Requirement(metadata_name)
def prepare_pep517_metadata(self):
assert self.pep517_backend is not None
metadata_dir = os.path.join(
self.setup_py_dir,
'pip-wheel-metadata'
)
ensure_dir(metadata_dir)
with self.build_env:
# Note that Pep517HookCaller implements a fallback for
# prepare_metadata_for_build_wheel, so we don't have to
# consider the possibility that this hook doesn't exist.
backend = self.pep517_backend
self.spin_message = "Preparing wheel metadata"
distinfo_dir = backend.prepare_metadata_for_build_wheel(
metadata_dir
)
self.metadata_directory = os.path.join(metadata_dir, distinfo_dir)
def run_egg_info(self):
if self.name:
logger.debug(
'Running setup.py (path:%s) egg_info for package %s',
self.setup_py, self.name,
)
else:
logger.debug(
'Running setup.py (path:%s) egg_info for package from %s',
self.setup_py, self.link,
)
script = SETUPTOOLS_SHIM % self.setup_py
base_cmd = [sys.executable, '-c', script]
if self.isolated:
base_cmd += ["--no-user-cfg"]
egg_info_cmd = base_cmd + ['egg_info']
# We can't put the .egg-info files at the root, because then the
# source code will be mistaken for an installed egg, causing
# problems
if self.editable:
egg_base_option = []
else:
egg_info_dir = os.path.join(self.setup_py_dir, 'pip-egg-info')
ensure_dir(egg_info_dir)
egg_base_option = ['--egg-base', 'pip-egg-info']
with self.build_env:
call_subprocess(
egg_info_cmd + egg_base_option,
cwd=self.setup_py_dir,
show_stdout=False,
command_desc='python setup.py egg_info')
@property
def egg_info_path(self):
if self._egg_info_path is None:
@ -556,13 +617,23 @@ class InstallRequirement(object):
return self._metadata
def get_dist(self):
"""Return a pkg_resources.Distribution built from self.egg_info_path"""
egg_info = self.egg_info_path.rstrip(os.path.sep)
base_dir = os.path.dirname(egg_info)
metadata = pkg_resources.PathMetadata(base_dir, egg_info)
dist_name = os.path.splitext(os.path.basename(egg_info))[0]
return pkg_resources.Distribution(
os.path.dirname(egg_info),
"""Return a pkg_resources.Distribution for this requirement"""
if self.metadata_directory:
base_dir, distinfo = os.path.split(self.metadata_directory)
metadata = pkg_resources.PathMetadata(
base_dir, self.metadata_directory
)
dist_name = os.path.splitext(distinfo)[0]
typ = pkg_resources.DistInfoDistribution
else:
egg_info = self.egg_info_path.rstrip(os.path.sep)
base_dir = os.path.dirname(egg_info)
metadata = pkg_resources.PathMetadata(base_dir, egg_info)
dist_name = os.path.splitext(os.path.basename(egg_info))[0]
typ = pkg_resources.Distribution
return typ(
base_dir,
project_name=dist_name,
metadata=metadata,
)

4
src/pip/_internal/resolve.py
View file

@ -35,7 +35,7 @@ class Resolver(object):
def __init__(self, preparer, session, finder, wheel_cache, use_user_site,
ignore_dependencies, ignore_installed, ignore_requires_python,
force_reinstall, isolated, upgrade_strategy):
force_reinstall, isolated, upgrade_strategy, use_pep517=None):
super(Resolver, self).__init__()
assert upgrade_strategy in self._allowed_strategies
@ -56,6 +56,7 @@ class Resolver(object):
self.ignore_installed = ignore_installed
self.ignore_requires_python = ignore_requires_python
self.use_user_site = use_user_site
self.use_pep517 = use_pep517
self._discovered_dependencies = defaultdict(list)
@ -273,6 +274,7 @@ class Resolver(object):
req_to_install,
isolated=self.isolated,
wheel_cache=self.wheel_cache,
use_pep517=self.use_pep517
)
parent_req_name = req_to_install.name
to_scan_again, add_to_parent = requirement_set.add_requirement(

49
src/pip/_internal/wheel.py
View file

@ -74,6 +74,14 @@ def open_for_csv(name, mode):
return open(name, mode + bin, **nl)
def replace_python_tag(wheelname, new_tag):
"""Replace the Python tag in a wheel file name with a new value.
"""
parts = wheelname.split('-')
parts[-3] = new_tag
return '-'.join(parts)
def fix_script(path):
"""Replace #!python with #!/path/to/python
Return True if file was changed."""
@ -677,7 +685,11 @@ class WheelBuilder(object):
def _build_one_inside_env(self, req, output_dir, python_tag=None):
with TempDirectory(kind="wheel") as temp_dir:
if self.__build_one(req, temp_dir.path, python_tag=python_tag):
if req.use_pep517:
builder = self._build_one_pep517
else:
builder = self._build_one_legacy
if builder(req, temp_dir.path, python_tag=python_tag):
try:
wheel_name = os.listdir(temp_dir.path)[0]
wheel_path = os.path.join(output_dir, wheel_name)
@ -702,10 +714,33 @@ class WheelBuilder(object):
SETUPTOOLS_SHIM % req.setup_py
] + list(self.global_options)
def __build_one(self, req, tempd, python_tag=None):
def _build_one_pep517(self, req, tempd, python_tag=None):
assert req.metadata_directory is not None
try:
req.spin_message = 'Building wheel for %s (PEP 517)' % (req.name,)
logger.debug('Destination directory: %s', tempd)
wheelname = req.pep517_backend.build_wheel(
tempd,
metadata_directory=req.metadata_directory
)
if python_tag:
# General PEP 517 backends don't necessarily support
# a "--python-tag" option, so we rename the wheel
# file directly.
newname = replace_python_tag(wheelname, python_tag)
os.rename(
os.path.join(tempd, wheelname),
os.path.join(tempd, newname)
)
return True
except Exception:
logger.error('Failed building wheel for %s', req.name)
return False
def _build_one_legacy(self, req, tempd, python_tag=None):
base_args = self._base_setup_args(req)
spin_message = 'Running setup.py bdist_wheel for %s' % (req.name,)
spin_message = 'Building wheel for %s (setup.py)' % (req.name,)
with open_spinner(spin_message) as spinner:
logger.debug('Destination directory: %s', tempd)
wheel_args = base_args + ['bdist_wheel', '-d', tempd] \
@ -744,6 +779,8 @@ class WheelBuilder(object):
"""
from pip._internal.models.link import Link
# TODO: This check fails if --no-cache-dir is set. And yet we
# might be able to build into the ephemeral cache, surely?
building_is_possible = self._wheel_dir or (
autobuilding and self.wheel_cache.cache_dir
)
@ -784,7 +821,7 @@ class WheelBuilder(object):
buildset.append((req, ephem_cache))
if not buildset:
return True
return []
# Build the wheels.
logger.info(
@ -856,5 +893,5 @@ class WheelBuilder(object):
'Failed to build %s',
' '.join([req.name for req in build_failure]),
)
# Return True if all builds were successful
return len(build_failure) == 0
# Return a list of requirements that failed to build
return build_failure

2
src/pip/_vendor/pep517/__init__.py
View file

@ -1,4 +1,4 @@
"""Wrappers to build Python packages using PEP 517 hooks
"""
__version__ = '0.2'
__version__ = '0.3'

10
src/pip/_vendor/pep517/_in_process.py
View file

@ -21,11 +21,17 @@ import sys
# This is run as a script, not a module, so it can't do a relative import
import compat
class BackendUnavailable(Exception):
"""Raised if we cannot import the backend"""
def _build_backend():
"""Find and load the build backend"""
ep = os.environ['PEP517_BUILD_BACKEND']
mod_path, _, obj_path = ep.partition(':')
obj = import_module(mod_path)
try:
obj = import_module(mod_path)
except ImportError:
raise BackendUnavailable
if obj_path:
for path_part in obj_path.split('.'):
obj = getattr(obj, path_part)
@ -173,6 +179,8 @@ def main():
json_out = {'unsupported': False, 'return_val': None}
try:
json_out['return_val'] = hook(**hook_input['kwargs'])
except BackendUnavailable:
json_out['no_backend'] = True
except GotUnsupportedOperation:
json_out['unsupported'] = True

33
src/pip/_vendor/pep517/wrappers.py
View file

@ -18,9 +18,20 @@ def tempdir():
finally:
shutil.rmtree(td)
class BackendUnavailable(Exception):
"""Will be raised if the backend cannot be imported in the hook process."""
class UnsupportedOperation(Exception):
"""May be raised by build_sdist if the backend indicates that it can't."""
def default_subprocess_runner(cmd, cwd=None, extra_environ=None):
"""The default method of calling the wrapper subprocess."""
env = os.environ.copy()
if extra_environ:
env.update(extra_environ)
check_call(cmd, cwd=cwd, env=env)
class Pep517HookCaller(object):
"""A wrapper around a source directory to be built with a PEP 517 backend.
@ -30,6 +41,16 @@ class Pep517HookCaller(object):
def __init__(self, source_dir, build_backend):
self.source_dir = abspath(source_dir)
self.build_backend = build_backend
self._subprocess_runner = default_subprocess_runner
# TODO: Is this over-engineered? Maybe frontends only need to
# set this when creating the wrapper, not on every call.
@contextmanager
def subprocess_runner(self, runner):
prev = self._subprocess_runner
self._subprocess_runner = runner
yield
self._subprocess_runner = prev
def get_requires_for_build_wheel(self, config_settings=None):
"""Identify packages required for building a wheel
@ -105,8 +126,6 @@ class Pep517HookCaller(object):
def _call_hook(self, hook_name, kwargs):
env = os.environ.copy()
# On Python 2, pytoml returns Unicode values (which is correct) but the
# environment passed to check_call needs to contain string values. We
# convert here by encoding using ASCII (the backend can only contain
@ -118,17 +137,21 @@ class Pep517HookCaller(object):
else:
build_backend = self.build_backend
env['PEP517_BUILD_BACKEND'] = build_backend
with tempdir() as td:
compat.write_json({'kwargs': kwargs}, pjoin(td, 'input.json'),
indent=2)
# Run the hook in a subprocess
check_call([sys.executable, _in_proc_script, hook_name, td],
cwd=self.source_dir, env=env)
self._subprocess_runner(
[sys.executable, _in_proc_script, hook_name, td],
cwd=self.source_dir,
extra_environ={'PEP517_BUILD_BACKEND': build_backend}
)
data = compat.read_json(pjoin(td, 'output.json'))
if data.get('unsupported'):
raise UnsupportedOperation
if data.get('no_backend'):
raise BackendUnavailable
return data['return_val']

2
src/pip/_vendor/vendor.txt
View file

@ -10,7 +10,7 @@ lockfile==0.12.2
progress==1.4
ipaddress==1.0.22 # Only needed on 2.6 and 2.7
packaging==18.0
pep517==0.2
pep517==0.3
pyparsing==2.2.1
pytoml==0.1.19
retrying==1.3.3

10
tests/conftest.py
View file

@ -56,14 +56,6 @@ def pytest_collection_modifyitems(config, items):
item.add_marker(pytest.mark.integration)
elif module_root_dir.startswith("unit"):
item.add_marker(pytest.mark.unit)
# We don't want to allow using the script resource if this is a
# unit test, as unit tests should not need all that heavy lifting
if set(getattr(item, "funcargnames", [])) & {"script"}:
raise RuntimeError(
"Cannot use the ``script`` funcarg in a unit test: "
"(filename = {}, item = {})".format(module_path, item)
)
else:
raise RuntimeError(
"Unknown test type (filename = {})".format(module_path)
@ -180,7 +172,7 @@ def pip_src(tmpdir_factory):
SRC_DIR,
pip_src.abspath,
ignore=shutil.ignore_patterns(
"*.pyc", "__pycache__", "contrib", "docs", "tasks", "*.txt",
"*.pyc", "__pycache__", "contrib", "docs", "tasks",
"tests", "pip.egg-info", "build", "dist", ".tox", ".git",
),
)

BIN
tests/data/backends/test_backend-0.1-py2.py3-none-any.whl Normal file
View file

Binary file not shown.

BIN
tests/data/backends/test_backend-0.1.tar.gz Normal file
View file

Binary file not shown.

1
tests/data/src/pep518_conflicting_requires/MANIFEST.in Normal file
View file

@ -0,0 +1 @@
include pyproject.toml

1
tests/data/src/pep518_conflicting_requires/pep518.py Normal file
View file

@ -0,0 +1 @@
#dummy

2
tests/data/src/pep518_conflicting_requires/pyproject.toml Normal file
View file

@ -0,0 +1,2 @@
[build-system]
requires = ["setuptools==1.0", "wheel"]

8
tests/data/src/pep518_conflicting_requires/setup.py Normal file
View file

@ -0,0 +1,8 @@
#!/usr/bin/env python
from setuptools import setup
setup(
name='pep518_conflicting_requires',
version='1.0.0',
py_modules=['pep518'],
)

5
tests/functional/test_completion.py
View file

@ -40,12 +40,13 @@ compctl -K _pip_completion pip"""),
COMPLETION_FOR_SUPPORTED_SHELLS_TESTS,
ids=[t[0] for t in COMPLETION_FOR_SUPPORTED_SHELLS_TESTS],
)
def test_completion_for_supported_shells(script, pip_src, shell, completion):
def test_completion_for_supported_shells(script, pip_src, common_wheels,
shell, completion):
"""
Test getting completion for bash shell
"""
# Re-install pip so we get the launchers.
script.pip_install_local('--no-build-isolation', pip_src)
script.pip_install_local('-f', common_wheels, pip_src)
result = script.pip('completion', '--' + shell, use_module=False)
assert completion in result.stdout, str(result.stdout)

53
tests/functional/test_install.py
View file

@ -12,9 +12,9 @@ from pip._internal.cli.status_codes import ERROR, SUCCESS
from pip._internal.models.index import PyPI, TestPyPI
from pip._internal.utils.misc import rmtree
from tests.lib import (
_create_svn_repo, _create_test_package, create_test_package_with_setup,
need_bzr, need_mercurial, path_to_url, pyversion, pyversion_tuple,
requirements_file,
_create_svn_repo, _create_test_package, create_basic_wheel_for_package,
create_test_package_with_setup, need_bzr, need_mercurial, path_to_url,
pyversion, pyversion_tuple, requirements_file,
)
from tests.lib.local_repos import local_checkout
from tests.lib.path import Path
@ -50,6 +50,20 @@ def test_pep518_build_env_uses_same_pip(script, data, pip_src, common_wheels):
)
def test_pep518_refuses_conflicting_requires(script, data):
create_basic_wheel_for_package(script, 'setuptools', '1.0')
create_basic_wheel_for_package(script, 'wheel', '1.0')
project_dir = data.src.join("pep518_conflicting_requires")
result = script.pip_install_local('-f', script.scratch_path,
project_dir, expect_error=True)
assert (
result.returncode != 0 and
('Some build dependencies for %s conflict with PEP 517/518 supported '
'requirements: setuptools==1.0 is incompatible with '
'setuptools>=40.2.0.' % path_to_url(project_dir)) in result.stderr
), str(result)
def test_pep518_refuses_invalid_requires(script, data, common_wheels):
result = script.pip(
'install', '-f', common_wheels,
@ -87,7 +101,17 @@ def test_pep518_allows_missing_requires(script, data, common_wheels):
def test_pep518_with_user_pip(script, pip_src, data, common_wheels):
script.pip("install", "--ignore-installed", "--user", pip_src)
"""
Check that build dependencies are installed into the build
environment without using build isolation for the pip invocation.
To ensure that we're not using build isolation when installing
the build dependencies, we install a user copy of pip in the
non-isolated environment, and break pip in the system site-packages,
so that isolated uses of pip will fail.
"""
script.pip("install", "--ignore-installed",
"-f", common_wheels, "--user", pip_src)
system_pip_dir = script.site_packages_path / 'pip'
system_pip_dir.rmtree()
system_pip_dir.mkdir()
@ -138,12 +162,13 @@ def test_pep518_forkbombs(script, data, common_wheels, command, package):
@pytest.mark.network
def test_pip_second_command_line_interface_works(script, data, pip_src):
def test_pip_second_command_line_interface_works(script, pip_src, data,
common_wheels):
"""
Check if ``pip<PYVERSION>`` commands behaves equally
"""
# Re-install pip so we get the launchers.
script.pip_install_local('--no-build-isolation', pip_src)
script.pip_install_local('-f', common_wheels, pip_src)
# On old versions of Python, urllib3/requests will raise a warning about
# the lack of an SSLContext.
kwargs = {}
@ -1136,10 +1161,10 @@ def test_install_builds_wheels(script, data, with_wheel):
for top, dirs, files in os.walk(wheels_cache):
wheels.extend(files)
# and built wheels for upper and wheelbroken
assert "Running setup.py bdist_wheel for upper" in str(res), str(res)
assert "Running setup.py bdist_wheel for wheelb" in str(res), str(res)
assert "Building wheel for upper" in str(res), str(res)
assert "Building wheel for wheelb" in str(res), str(res)
# Wheels are built for local directories, but not cached.
assert "Running setup.py bdist_wheel for requir" in str(res), str(res)
assert "Building wheel for requir" in str(res), str(res)
# wheelbroken has to run install
# into the cache
assert wheels != [], str(res)
@ -1165,11 +1190,11 @@ def test_install_no_binary_disables_building_wheels(script, data, with_wheel):
# Must have installed it all
assert expected in str(res), str(res)
# and built wheels for wheelbroken only
assert "Running setup.py bdist_wheel for wheelb" in str(res), str(res)
assert "Building wheel for wheelb" in str(res), str(res)
# Wheels are built for local directories, but not cached across runs
assert "Running setup.py bdist_wheel for requir" in str(res), str(res)
assert "Building wheel for requir" in str(res), str(res)
# Don't build wheel for upper which was blacklisted
assert "Running setup.py bdist_wheel for upper" not in str(res), str(res)
assert "Building wheel for upper" not in str(res), str(res)
# Wheels are built for local directories, but not cached across runs
assert "Running setup.py install for requir" not in str(res), str(res)
# And these two fell back to sdist based installed.
@ -1188,7 +1213,7 @@ def test_install_no_binary_disables_cached_wheels(script, data, with_wheel):
'upper', expect_stderr=True)
assert "Successfully installed upper-2.0" in str(res), str(res)
# No wheel building for upper, which was blacklisted
assert "Running setup.py bdist_wheel for upper" not in str(res), str(res)
assert "Building wheel for upper" not in str(res), str(res)
# Must have used source, not a cached wheel to install upper.
assert "Running setup.py install for upper" in str(res), str(res)
@ -1204,7 +1229,7 @@ def test_install_editable_with_wrong_egg_name(script):
result = script.pip(
'install', '--editable', 'file://%s#egg=pkgb' % pkga_path,
expect_error=True)
assert ("egg_info for package pkgb produced metadata "
assert ("Generating metadata for package pkgb produced metadata "
"for project name pkga. Fix your #egg=pkgb "
"fragments.") in result.stderr
assert "Successfully installed pkga" in str(result), str(result)

2
tests/functional/test_install_config.py
View file

@ -216,6 +216,6 @@ def test_install_no_binary_via_config_disables_cached_wheels(
os.unlink(config_file.name)
assert "Successfully installed upper-2.0" in str(res), str(res)
# No wheel building for upper, which was blacklisted
assert "Running setup.py bdist_wheel for upper" not in str(res), str(res)
assert "Building wheel for upper" not in str(res), str(res)
# Must have used source, not a cached wheel to install upper.
assert "Running setup.py install for upper" in str(res), str(res)

96
tests/functional/test_pep517.py
View file

@ -4,26 +4,108 @@ from pip._internal.build_env import BuildEnvironment
from pip._internal.download import PipSession
from pip._internal.index import PackageFinder
from pip._internal.req import InstallRequirement
from tests.lib import path_to_url
def make_project(tmpdir, requires=[], backend=None):
project_dir = (tmpdir / 'project').mkdir()
buildsys = {'requires': requires}
if backend:
buildsys['build-backend'] = backend
data = pytoml.dumps({'build-system': buildsys})
tmpdir.join('pyproject.toml').write(data)
return tmpdir
project_dir.join('pyproject.toml').write(data)
return project_dir
def test_backend(tmpdir, data):
"""Can we call a requirement's backend successfully?"""
project = make_project(tmpdir, backend="dummy_backend")
req = InstallRequirement(None, None, source_dir=project)
"""Check we can call a requirement's backend successfully"""
project_dir = make_project(tmpdir, backend="dummy_backend")
req = InstallRequirement(None, None, source_dir=project_dir)
req.load_pyproject_toml()
env = BuildEnvironment()
finder = PackageFinder([data.backends], [], session=PipSession())
env.install_requirements(finder, ["dummy_backend"], "Installing")
assert not env.missing_requirements(["dummy_backend"])
env.install_requirements(finder, ["dummy_backend"], 'normal', "Installing")
conflicting, missing = env.check_requirements(["dummy_backend"])
assert not conflicting and not missing
assert hasattr(req.pep517_backend, 'build_wheel')
with env:
assert req.pep517_backend.build_wheel("dir") == "Backend called"
def test_pep517_install(script, tmpdir, data):
"""Check we can build with a custom backend"""
project_dir = make_project(
tmpdir, requires=['test_backend'],
backend="test_backend"
)
result = script.pip(
'install', '--no-index', '-f', data.backends, project_dir
)
result.assert_installed('project', editable=False)
def test_pep517_install_with_reqs(script, tmpdir, data):
"""Backend generated requirements are installed in the build env"""
project_dir = make_project(
tmpdir, requires=['test_backend'],
backend="test_backend"
)
project_dir.join("backend_reqs.txt").write("simplewheel")
result = script.pip(
'install', '--no-index',
'-f', data.backends,
'-f', data.packages,
project_dir
)
result.assert_installed('project', editable=False)
def test_no_use_pep517_without_setup_py(script, tmpdir, data):
"""Using --no-use-pep517 requires setup.py"""
project_dir = make_project(
tmpdir, requires=['test_backend'],
backend="test_backend"
)
result = script.pip(
'install', '--no-index', '--no-use-pep517',
'-f', data.backends,
project_dir,
expect_error=True
)
assert 'project does not have a setup.py' in result.stderr
def test_conflicting_pep517_backend_requirements(script, tmpdir, data):
project_dir = make_project(
tmpdir, requires=['test_backend', 'simplewheel==1.0'],
backend="test_backend"
)
project_dir.join("backend_reqs.txt").write("simplewheel==2.0")
result = script.pip(
'install', '--no-index',
'-f', data.backends,
'-f', data.packages,
project_dir,
expect_error=True
)
assert (
result.returncode != 0 and
('Some build dependencies for %s conflict with the backend '
'dependencies: simplewheel==1.0 is incompatible with '
'simplewheel==2.0.' % path_to_url(project_dir)) in result.stderr
), str(result)
def test_pep517_backend_requirements_already_satisfied(script, tmpdir, data):
project_dir = make_project(
tmpdir, requires=['test_backend', 'simplewheel==1.0'],
backend="test_backend"
)
project_dir.join("backend_reqs.txt").write("simplewheel")
result = script.pip(
'install', '--no-index',
'-f', data.backends,
'-f', data.packages,
project_dir,
)
assert 'Installing backend dependencies:' not in result.stdout

2
tests/functional/test_wheel.py
View file

@ -65,7 +65,7 @@ def test_pip_wheel_builds_when_no_binary_set(script, data):
'wheel', '--no-index', '--no-binary', ':all:',
'-f', data.find_links,
'simple==3.0')
assert "Running setup.py bdist_wheel for simple" in str(res), str(res)
assert "Building wheel for simple" in str(res), str(res)
def test_pip_wheel_builds_editable_deps(script, data):

8
tests/lib/__init__.py
View file

@ -687,9 +687,15 @@ def create_test_package_with_setup(script, **setup_kwargs):
return pkg_path
def create_basic_wheel_for_package(script, name, version, depends, extras):
def create_basic_wheel_for_package(script, name, version,
depends=None, extras=None):
if depends is None:
depends = []
if extras is None:
extras = {}
files = {
"{name}/__init__.py": """
__version__ = {version}
def hello():
return "Hello From {name}"
""",

194
tests/unit/test_build_env.py Normal file
View file

@ -0,0 +1,194 @@
from textwrap import dedent
import pytest
from pip._internal.build_env import BuildEnvironment
from pip._internal.download import PipSession
from pip._internal.index import PackageFinder
from tests.lib import create_basic_wheel_for_package
def indent(text, prefix):
return '\n'.join((prefix if line else '') + line
for line in text.split('\n'))
def run_with_build_env(script, setup_script_contents,
test_script_contents=None):
build_env_script = script.scratch_path / 'build_env.py'
build_env_script.write(
dedent(
'''
from __future__ import print_function
import subprocess
import sys
from pip._internal.build_env import BuildEnvironment
from pip._internal.download import PipSession
from pip._internal.index import PackageFinder
finder = PackageFinder([%r], [], session=PipSession())
build_env = BuildEnvironment()
try:
''' % str(script.scratch_path)) +
indent(dedent(setup_script_contents), ' ') +
dedent(
'''
if len(sys.argv) > 1:
with build_env:
subprocess.check_call((sys.executable, sys.argv[1]))
finally:
build_env.cleanup()
''')
)
args = ['python', build_env_script]
if test_script_contents is not None:
test_script = script.scratch_path / 'test.py'
test_script.write(dedent(test_script_contents))
args.append(test_script)
return script.run(*args)
def test_build_env_allow_empty_requirements_install():
build_env = BuildEnvironment()
for prefix in ('normal', 'overlay'):
build_env.install_requirements(None, [], prefix, None)
def test_build_env_allow_only_one_install(script):
create_basic_wheel_for_package(script, 'foo', '1.0')
create_basic_wheel_for_package(script, 'bar', '1.0')
finder = PackageFinder([script.scratch_path], [], session=PipSession())
build_env = BuildEnvironment()
for prefix in ('normal', 'overlay'):
build_env.install_requirements(finder, ['foo'], prefix,
'installing foo in %s' % prefix)
with pytest.raises(AssertionError):
build_env.install_requirements(finder, ['bar'], prefix,
'installing bar in %s' % prefix)
with pytest.raises(AssertionError):
build_env.install_requirements(finder, [], prefix,
'installing in %s' % prefix)
def test_build_env_requirements_check(script):
create_basic_wheel_for_package(script, 'foo', '2.0')
create_basic_wheel_for_package(script, 'bar', '1.0')
create_basic_wheel_for_package(script, 'bar', '3.0')
create_basic_wheel_for_package(script, 'other', '0.5')
script.pip_install_local('-f', script.scratch_path, 'foo', 'bar', 'other')
run_with_build_env(
script,
'''
r = build_env.check_requirements(['foo', 'bar', 'other'])
assert r == (set(), {'foo', 'bar', 'other'}), repr(r)
r = build_env.check_requirements(['foo>1.0', 'bar==3.0'])
assert r == (set(), {'foo>1.0', 'bar==3.0'}), repr(r)
r = build_env.check_requirements(['foo>3.0', 'bar>=2.5'])
assert r == (set(), {'foo>3.0', 'bar>=2.5'}), repr(r)
''')
run_with_build_env(
script,
'''
build_env.install_requirements(finder, ['foo', 'bar==3.0'], 'normal',
'installing foo in normal')
r = build_env.check_requirements(['foo', 'bar', 'other'])
assert r == (set(), {'other'}), repr(r)
r = build_env.check_requirements(['foo>1.0', 'bar==3.0'])
assert r == (set(), set()), repr(r)
r = build_env.check_requirements(['foo>3.0', 'bar>=2.5'])
assert r == ({('foo==2.0', 'foo>3.0')}, set()), repr(r)
''')
run_with_build_env(
script,
'''
build_env.install_requirements(finder, ['foo', 'bar==3.0'], 'normal',
'installing foo in normal')
build_env.install_requirements(finder, ['bar==1.0'], 'overlay',
'installing foo in overlay')
r = build_env.check_requirements(['foo', 'bar', 'other'])
assert r == (set(), {'other'}), repr(r)
r = build_env.check_requirements(['foo>1.0', 'bar==3.0'])
assert r == ({('bar==1.0', 'bar==3.0')}, set()), repr(r)
r = build_env.check_requirements(['foo>3.0', 'bar>=2.5'])
assert r == ({('bar==1.0', 'bar>=2.5'), ('foo==2.0', 'foo>3.0')}, \
set()), repr(r)
''')
def test_build_env_overlay_prefix_has_priority(script):
create_basic_wheel_for_package(script, 'pkg', '2.0')
create_basic_wheel_for_package(script, 'pkg', '4.3')
result = run_with_build_env(
script,
'''
build_env.install_requirements(finder, ['pkg==2.0'], 'overlay',
'installing pkg==2.0 in overlay')
build_env.install_requirements(finder, ['pkg==4.3'], 'normal',
'installing pkg==4.3 in normal')
''',
'''
from __future__ import print_function
print(__import__('pkg').__version__)
''')
assert result.stdout.strip() == '2.0', str(result)
def test_build_env_isolation(script):
# Create dummy `pkg` wheel.
pkg_whl = create_basic_wheel_for_package(script, 'pkg', '1.0')
# Install it to site packages.
script.pip_install_local(pkg_whl)
# And a copy in the user site.
script.pip_install_local('--ignore-installed', '--user', pkg_whl)
# And to another directory available through a .pth file.
target = script.scratch_path / 'pth_install'
script.pip_install_local('-t', target, pkg_whl)
(script.site_packages_path / 'build_requires.pth').write(
str(target) + '\n'
)
# And finally to yet another directory available through PYTHONPATH.
target = script.scratch_path / 'pypath_install'
script.pip_install_local('-t', target, pkg_whl)
script.environ["PYTHONPATH"] = target
run_with_build_env(
script, '',
r'''
from __future__ import print_function
from distutils.sysconfig import get_python_lib
import sys
try:
import pkg
except ImportError:
pass
else:
print('imported `pkg` from `%s`' % pkg.__file__, file=sys.stderr)
print('system sites:\n ' + '\n '.join(sorted({
get_python_lib(plat_specific=0),
get_python_lib(plat_specific=1),
})), file=sys.stderr)
print('sys.path:\n ' + '\n '.join(sys.path), file=sys.stderr)
sys.exit(1)
''')

2
tests/unit/test_req.py
View file

@ -630,7 +630,7 @@ def test_mismatched_versions(caplog, tmpdir):
shutil.copytree(original_source, source_dir)
req = InstallRequirement(req=Requirement('simplewheel==2.0'),
comes_from=None, source_dir=source_dir)
req.run_egg_info()
req.prepare_metadata()
req.assert_source_matches_version()
assert caplog.records[-1].message == (
'Requested simplewheel==2.0, '

15
tests/unit/test_wheel.py
View file

@ -90,6 +90,21 @@ def test_wheel_version(tmpdir, data):
assert not wheel.wheel_version(tmpdir + 'broken')
def test_python_tag():
wheelnames = [
'simplewheel-1.0-py2.py3-none-any.whl',
'simplewheel-1.0-py27-none-any.whl',
'simplewheel-2.0-1-py2.py3-none-any.whl',
]
newnames = [
'simplewheel-1.0-py37-none-any.whl',
'simplewheel-1.0-py37-none-any.whl',
'simplewheel-2.0-1-py37-none-any.whl',
]
for name, new in zip(wheelnames, newnames):
assert wheel.replace_python_tag(name, 'py37') == new
def test_check_compatibility():
name = 'test'
vc = wheel.VERSION_COMPATIBLE

2
tools/travis/run.sh
View file

@ -41,7 +41,7 @@ echo "TOXENV=${TOXENV}"
set -x
if [[ "$GROUP" == "1" ]]; then
# Unit tests
tox -- -m unit
tox -- --use-venv -m unit
# Integration tests (not the ones for 'pip install')
tox -- --use-venv -m integration -n 4 --duration=5 -k "not test_install"
elif [[ "$GROUP" == "2" ]]; then