One of the downsides of pip's new hiding of build chatter is that for
packages that take a very long time to build (e.g. scipy) the user gets
no indication that anything is happening for a very long time (e.g. tens
of minutes), and is likely to get frustrated and hit Control-C. This can
also create issues for automated systems that kill jobs that don't
produce occasional output (e.g. Travis-CI).
This commit implements an idea discussed here:
https://github.com/pypa/pip/issues/2732#issuecomment-153215371
where we put up a spinner that rotates whenever the underlying build
produces output. I tried it on scipy, and it the experience was quite
pleasant! It spun around, sometimes fast and sometimes slow, and then
there was one uncomfortable pause for ~1 minute while a very gnarly C++
file got compiled, but that's okay because it was actually providing
accurate feedback.
It looks like:
```
Running setup.py install for scipy ... /
Running setup.py install for scipy ... -
Running setup.py install for scipy ... \
Running setup.py install for scipy ... done
```
or if the command has non-zero return code, or an exception is raised,
you get:
```
Running setup.py install for scipy ... /
Running setup.py install for scipy ... -
Running setup.py install for scipy ... \
Running setup.py install for scipy ... error
```
* Add --require-hashes option. This is handy in deployment scripts to force application authors to hash their requirements. It is also a convenient way to get pip to show computed hashes for a virgin, unhashed requirements file. Eventually, additions to `pip freeze` should fill a superset of this use case.
* In --require-hashes mode, at least one hash is required to match for each requirement.
* Option-based requirements (--sha256=...) turn on --require-hashes mode implicitly.
* Internet-derived URL-based hashes are "necessary but not sufficient": they do not satisfy --require-hashes mode when they match, but they are still used to guard against transmission errors.
* Other URL-based requirements (#md5=...) are treated just like flag-based ones, except they don't turn on --require-hashes.
* Complain informatively, with the most devastating errors first so you don't chase your tail all day only to run up against a brick wall at the end. This also means we don't complain that a hash is missing, only for the user to find, after fixing it, that we have no idea how to even compute a hash for that type of requirement.
* Complain about unpinned requirements when hash-checking mode is on, lest they cause the user surprise later.
* Complain about missing hashes.
* Complain about requirement types we don't know how to hash (like VCS ones and local dirs).
* Have InstallRequirement keep its original Link around (original_link) so we can differentiate between URL hashes from requirements files and ones downloaded from the (untrustworthy) internet.
* Remove test_download_hashes, which is obsolete. Similar coverage is provided in test_utils.TestHashes and the various hash cases in test_req.py.
This fixes 2 aspects of `pip install output`:
1. When `pip install` succeeds, it's still printing a lot of output from
the package's setup.py. The average consumer of Python packages, when
they do `pip install lxml`, probably doesn't care to see a bunch of
output about:
- copying files to a `build` directory
- installing and running Cython
- compiling C code
This is noise to most when the `pip install` succeeds. It's useful to
see all the output when the install fails, which is the subject of #2
below. On success, the output is now very clean with 5 short lines:
$ pip install lxml
Collecting lxml
Using cached lxml-3.4.2.tar.gz
Installing collected packages: lxml
Running setup.py install for lxml
Successfully installed lxml-3.4.2
2. When there's an error from `pip install`, it's annoying to have to
scroll through 2 different copies of the failure output (especially when
one is filtered and one is unfiltered so one might have stuff that the
other doesn't). This makes it not print the filtered version so that
there is just the unfiltered version and nothing is repeated.
$ pip install ~/dev/git-repos/lxml
Processing /Users/marca/dev/git-repos/lxml
Installing collected packages: lxml
Running setup.py install for lxml
Complete output from command ...
cc -c /var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/xmlCheckVersion4tBaVV.c -o var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/xmlCheckVersion4tBaVV.o
/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/xmlCheckVersion4tBaVV.c:1:10: fatal error: 'libxml/xmlversion.h' file not found
#include "libxml/xmlversion.h"
^
1 error generated.
----------------------------------------
Command "/Users/marca/python/virtualenvs/pip/bin/python -c ...
None of the lines above are repeated.
Fixes a long-time annoyance that setup.py failure output has a blank
line after every line -- e.g.:
$ pip wheel ~/dev/git-repos/lxml
Processing /Users/marca/dev/git-repos/lxml
...
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/libxml2 -Isrc/lxml/includes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.6-intel-2.7/src/lxml/lxml.etree.o -w -flat_namespace
In file included from src/lxml/lxml.etree.c:314:
src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
#include "libxml/xmlversion.h"
^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1
...
- Move code into separate method - check_required_packages
- Use new import_or_raise function - mostly so we don't have to have
hacks for flake8 and comments to explain said hacks
This already got stripped just fine for non-wheel installs, but for
wheels, it only stripped the lined where `SyntaxError` was present
leaving the rest of the stacktrace.
