This refactor is intended to make it easier to make configuration related improvements in the future, since all the code is in a separated unit.
The unit tests have been updated partially to merely update them. They now patch underscore names which means that they probably need updating.
to fetch tags in addition to everything else that
is normally fetched; this is necessary in case a git requirement url
points to a tag or commit that is not on a branch
With the --platform option, a user can download wheels with
a different platform than that of the local machine running the command.
With the --python-version option, a user can
download wheels that are explicitly compatible with a specific
Python interpreter version.
This functionality is meant for utilities that gather dependencies
and prepare distributions for other platforms.
Some systems have /tmp symlinked which confuses custom builds, such as
numpy. This ensures that real path is passed and that such builds
resolve their paths correctly during build and install.
Added test for the change and also for the previous related fix: #707
---
*This was migrated from pypa/pip#3079 to reparent it to the ``master``
branch. Please see original pull request for any previous discussion.*
`pip install -r reqs.txt` was failing when the requirements file includes
an unsupported wheel, regardless of whether it is conditionally removed
by a marker. This patch fixes that issue.
Additionally, this patch makes pip check local file wheels for
compatibility. Previously, a requirements file could include a path
to a valid wheel for any platform and pip would happily install it.
Downstream forks/redistributors of glibc apparently sometimes add junk
onto the end of the version number -- e.g. Linaro glibc might have a
version number like "2.20-2014.11". This makes the glibc version number
parsing code ignore the rest of the string after the minor version
number, to be robust against Linaro's current usage, as well as whatever
exciting new weirdnesses that other redistributors might come up with
in the future.
Fixes: gh-3588
PipSession attempts to create a user_agent which results in calls to
platform.linux_distribution being called, there is a chance this hits
platform._dist_try_harder, which calls os.path.exists("/a/bad/path")
then attempts to open and read from said path resulting in brekage.
This creates PipSession before mock patches os.path.exists then only
patches the call that requires os.path.exists to be patched.
Removed the mention of "package index options" in the docs, because they don't all fit that category anymore. Not even --no-binary and --only-binary do; they're "install options".
are unavailable, but issue a warning if this is used.
2. Explicitly handle the case where the unicode detection finds wide
unicode but this is a 3.3+ build (necessary due to #1)
3. Fix tests broken due to #2.
For dependencies that are properly pinned and hashed (not really dependencies at all, if you like, since they're explicit, root-level requirements), we install them as normal. For ones that are not pinned and hashes, we raise the errors typical of any unhashed requirement in --require-hashes mode.
Since the stanza under "if not ignore_dependencies" doesn't actually add anything if it's already in the RequirementSet, not much has to be done in the way of code: the unhashed deps don't have any hashes, so we complain about them as per usual.
Also...
* Revise wording of HashUnpinned errors. They can be raised even if no hash is specified, so the previous wording was misleading.
* Make wording of HashMissing less awkward.
dstufft is nervous about blowing a single-char option on something that will usually be copied and pasted anyway. We can always put it back later if it proves to be a pain.
* Add --require-hashes option. This is handy in deployment scripts to force application authors to hash their requirements. It is also a convenient way to get pip to show computed hashes for a virgin, unhashed requirements file. Eventually, additions to `pip freeze` should fill a superset of this use case.
* In --require-hashes mode, at least one hash is required to match for each requirement.
* Option-based requirements (--sha256=...) turn on --require-hashes mode implicitly.
* Internet-derived URL-based hashes are "necessary but not sufficient": they do not satisfy --require-hashes mode when they match, but they are still used to guard against transmission errors.
* Other URL-based requirements (#md5=...) are treated just like flag-based ones, except they don't turn on --require-hashes.
* Complain informatively, with the most devastating errors first so you don't chase your tail all day only to run up against a brick wall at the end. This also means we don't complain that a hash is missing, only for the user to find, after fixing it, that we have no idea how to even compute a hash for that type of requirement.
* Complain about unpinned requirements when hash-checking mode is on, lest they cause the user surprise later.
* Complain about missing hashes.
* Complain about requirement types we don't know how to hash (like VCS ones and local dirs).
* Have InstallRequirement keep its original Link around (original_link) so we can differentiate between URL hashes from requirements files and ones downloaded from the (untrustworthy) internet.
* Remove test_download_hashes, which is obsolete. Similar coverage is provided in test_utils.TestHashes and the various hash cases in test_req.py.
We purposely keep it off the CLI for now. optparse isn't really geared to expose interspersed args and options, so a more heavy-handed approach will be necessary to support things like `pip install SomePackage --sha256=abcdef... OtherPackage --sha256=012345...`.
This saves a network hop when using git and passing an explicit sha
as a ref by comparing the version that's already checked out.
Yields a ~4x speedup on my local machine
Before:
```
$ /usr/local/bin/pip --version
pip 7.1.0 from /usr/local/lib/python2.7/site-packages (python 2.7)
$ time /usr/local/bin/pip install --disable-pip-version-check -e git+https://github.com/getsentry/raven-python.git@56fc6f7beecf445843d0ec7052bb8c6f0ea80a2e#egg=raven_dev
Obtaining raven-dev from git+https://github.com/getsentry/raven-python.git@56fc6f7beecf445843d0ec7052bb8c6f0ea80a2e#egg=raven_dev
Updating ./src/raven-dev clone (to 56fc6f7beecf445843d0ec7052bb8c6f0ea80a2e)
Could not find a tag or branch '56fc6f7beecf445843d0ec7052bb8c6f0ea80a2e', assuming commit.
Installing collected packages: raven-dev
Running setup.py develop for raven-dev
Successfully installed raven-dev
/usr/local/bin/pip install --disable-pip-version-check -e 0.84s user 0.48s system 39% cpu 3.300 total
```
After:
```
$ /Users/matt/.virtualenvs/pip/bin/pip --version
pip 7.2.0.dev0 from /Users/matt/code/pip (python 2.7)
$ time /Users/matt/.virtualenvs/pip/bin/pip install --disable-pip-version-check -e git+https://github.com/getsentry/raven-python.git@56fc6f7beecf445843d0ec7052bb8c6f0ea80a2e#egg=raven_dev
Obtaining raven-dev from git+https://github.com/getsentry/raven-python.git@56fc6f7beecf445843d0ec7052bb8c6f0ea80a2e#egg=raven_dev
checking version
Skipping because already up-to-date.
Installing collected packages: raven-dev
Running setup.py develop for raven-dev
Successfully installed raven-dev
/Users/matt/.virtualenvs/pip/bin/pip install --disable-pip-version-check -e 0.59s user 0.22s system 98% cpu 0.824 total
```
Change 3affcaa2b8 attempts to reset
purelib & platlib to any "install-dir" specified by the user in
setup.cfg. This code is used when we are installing wheels.
The problem with this is that distutils is *always* setting
"i.install_lib" -- even when the user specifies nothing. This has the
result of unconditionally setting purelib == platlib.
On some systems this results in .so's from the wheel getting installed
into /usr/lib/python2.7 (purelib) rather than /usr/lib64/python
(platlib). Because distribution-packaged libraries have installed
their .so into platlib, we can now have a situation where the new
pip-installed library is picking up an old .so from the distro package
... with predictably bad results.
This takes the approach of checking the configuration to see if the
user has overridden install-dir and only resetting the paths if they
have. The override case is covered by existing test-cases.
Closes#2940
This adds constraints files. Like requirements files constraints files
control what version of a package is installed, but unlike
requirements files this doesn't itself choose to install the package.
This allows things that aren't explicitly desired to be constrained if
and only if they are installed.
We only want to prompt people to upgrade their pip version if the
newer version is not a post release of the version they have and
it's not a pre-release version.
and adjust the logic to match; the result is simpler.
2) Due to #1, we can remove some hairy "format_control" hacks
3) Due to #1, we have to relax the parsing and allow:
- multiple options per line
- any supported option on a line with a requirement (not just
--install-option/--global-option, although they are the only
options that are passed into a requirement)
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
With wheel autobuilding in place a release blocker is some granular
way to opt-out of wheels for known-bad packages. This patch introduces
two new options: --no-binary and --only-binary to control what
archives we are willing to use on both a global and per-package basis.
This also closes#2084
Wheel cache lookups become more complex when we wish to allow binary
blacklisting. Rather than passing more parameters around, replace
cache_root with wheel_cache, and create a wheel cache in all the
relevant command entry points.
two major changes:
1) re-use the optparse options in pip.cmdoptions instead of maintaining
a custom parser
2) as a result of #1, simplify the call stack
from: parse_requirements -> parse_content -> parse_line
to: parse_requirements -> process_line
beyond #1/#2, minor cosmetics and adjusting the tests to match
This allows lines such as the following to exist in requirements files:
INITools==0.2 --install-options="--prefix=/opt"
virtualenv>=1 --global-options="--no-user-cfg"
In addition, the requirements file parser was overhauled with simplicity
and clarity in mind.
We we normalising it right before making it, after passing it
RequirementSet, which means RequirementSet also had to normalise it.
And then we passed it to WheelBuilder which as a result had to
normalise it too.
Instead, pass it as-is to RequirementSet, have that normalise it, and
then pull it back out in WheelBuilder.
This speeds up the _filter_install function that is used to filter the
output of `python setup.py install` when installing packages. It does
this by using a single regex which is pre-compiled and thus avoiding a
`for` loop over 15 different regexes.
Before:
$ python
Python 2.7.9 (v2.7.9:648dcafa7e5f, Dec 10 2014, 10:10:46)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> timeit.timeit("""_filter_install("if we've already set distribute as a conflict to setuptools blah blah blah blah")""", setup='from pip.req.req_install import InstallRequirement; ir = InstallRequirement("foo", None, None); _filter_install = ir._filter_install')
21.220640897750854
After:
$ python
Python 2.7.9 (v2.7.9:648dcafa7e5f, Dec 10 2014, 10:10:46)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> timeit.timeit("""_filter_install("if we've already set distribute as a conflict to setuptools blah blah blah blah")""", setup='from pip.req.req_install import InstallRequirement; ir = InstallRequirement("foo", None, None); _filter_install = ir._filter_install')
0.9454500675201416
If an exact version is specified for install, and that version is
already installed, then there is no point going to pypi as no install
is needed.
Adds a test called
`test_upgrade_no_look_at_pypi_if_exact_version_installed`.
This is a rework of PR #771, because that PR is old and has merge
conflicts that were easier to fix by applying the changes manually.
