2013-08-16 18:57:34 +02:00
|
|
|
Policy
|
|
|
|
======
|
|
|
|
|
2016-01-19 14:52:21 +01:00
|
|
|
* Vendored libraries **MUST** not be modified except as required to
|
|
|
|
successfully vendor them.
|
|
|
|
|
|
|
|
* Vendored libraries **MUST** be released copies of libraries available on
|
|
|
|
PyPI.
|
|
|
|
|
|
|
|
* The versions of libraries vendored in pip **MUST** be reflected in
|
|
|
|
``pip/_vendor/vendor.txt``.
|
|
|
|
|
2016-11-23 16:37:19 +01:00
|
|
|
* Vendored libraries **MUST** function without any build steps such as ``2to3`` or
|
2016-01-19 14:52:21 +01:00
|
|
|
compilation of C code, pratically this limits to single source 2.x/3.x and
|
|
|
|
pure Python.
|
|
|
|
|
|
|
|
* Any modifications made to libraries **MUST** be noted in
|
2016-11-06 23:34:10 +01:00
|
|
|
``pip/_vendor/README.rst`` and their corresponding patches **MUST** be
|
|
|
|
included ``tasks/vendoring/patches``.
|
2016-01-19 12:57:17 +01:00
|
|
|
|
|
|
|
|
|
|
|
Rationale
|
|
|
|
---------
|
|
|
|
|
2018-07-15 15:07:45 +02:00
|
|
|
Historically pip has not had any dependencies except for ``setuptools`` itself,
|
2016-01-19 12:57:17 +01:00
|
|
|
choosing instead to implement any functionality it needed to prevent needing
|
2018-07-15 15:07:45 +02:00
|
|
|
a dependency. However, starting with pip 1.5, we began to replace code that was
|
2016-01-19 12:57:17 +01:00
|
|
|
implemented inside of pip with reusable libraries from PyPI. This brought the
|
|
|
|
typical benefits of reusing libraries instead of reinventing the wheel like
|
|
|
|
higher quality and more battle tested code, centralization of bug fixes
|
|
|
|
(particularly security sensitive ones), and better/more features for less work.
|
|
|
|
|
|
|
|
However, there is several issues with having dependencies in the traditional
|
|
|
|
way (via ``install_requires``) for pip. These issues are:
|
|
|
|
|
2016-11-23 16:37:19 +01:00
|
|
|
* **Fragility.** When pip depends on another library to function then if for
|
2016-01-19 12:57:17 +01:00
|
|
|
whatever reason that library either isn't installed or an incompatible
|
|
|
|
version is installed then pip ceases to function. This is of course true for
|
|
|
|
all Python applications, however for every application *except* for pip the
|
2016-11-23 16:37:19 +01:00
|
|
|
way you fix it is by re-running pip. Obviously, when pip can't run, you can't
|
|
|
|
use pip to fix pip, so you're left having to manually resolve dependencies and
|
2016-01-19 12:57:17 +01:00
|
|
|
installing them by hand.
|
|
|
|
|
2016-11-23 16:37:19 +01:00
|
|
|
* **Making other libraries uninstallable.** One of pip's current dependencies is
|
|
|
|
the ``requests`` library, for which pip requires a fairly recent version to run.
|
2018-07-15 15:07:45 +02:00
|
|
|
If pip depended on ``requests`` in the traditional manner, then we'd either
|
2017-12-16 13:35:37 +01:00
|
|
|
have to maintain compatibility with every ``requests`` version that has ever
|
2016-11-23 16:37:19 +01:00
|
|
|
existed (and ever will), OR allow pip to render certain versions of ``requests``
|
2017-12-16 13:35:37 +01:00
|
|
|
uninstallable. (The second issue, although technically true for any Python
|
|
|
|
application, is magnified by pip's ubiquity; pip is installed by default in
|
2016-11-23 16:37:19 +01:00
|
|
|
Python, in ``pyvenv``, and in ``virtualenv``.)
|
|
|
|
|
2017-12-16 13:35:37 +01:00
|
|
|
* **Security.** This might seem puzzling at first glance, since vendoring
|
2016-11-23 16:37:19 +01:00
|
|
|
has a tendency to complicate updating dependencies for security updates,
|
2017-12-16 13:35:37 +01:00
|
|
|
and that holds true for pip. However, given the *other* reasons for avoiding
|
|
|
|
dependencies, the alternative is for pip to reinvent the wheel itself.
|
|
|
|
This is what pip did historically. It forced pip to re-implement its own
|
|
|
|
HTTPS verification routines as a workaround for the Python standard library's
|
|
|
|
lack of SSL validation, which resulted in similar bugs in the validation routine
|
2016-11-23 16:37:19 +01:00
|
|
|
in ``requests`` and ``urllib3``, except that they had to be discovered and
|
2017-12-16 13:35:37 +01:00
|
|
|
fixed independently. Even though we're vendoring, reusing libraries keeps pip
|
2016-11-23 16:37:19 +01:00
|
|
|
more secure by relying on the great work of our dependencies, *and* allowing for
|
|
|
|
faster, easier security fixes by simply pulling in newer versions of dependencies.
|
|
|
|
|
|
|
|
* **Bootstrapping.** Currently most popular methods of installing pip rely
|
2017-12-16 13:35:37 +01:00
|
|
|
on pip's self-contained nature to install pip itself. These tools work by bundling
|
|
|
|
a copy of pip, adding it to ``sys.path``, and then executing that copy of pip.
|
|
|
|
This is done instead of implementing a "mini installer" (to reduce duplication);
|
|
|
|
pip already knows how to install a Python package, and is far more battle-tested
|
2016-11-23 16:37:19 +01:00
|
|
|
than any "mini installer" could ever possibly be.
|
|
|
|
|
|
|
|
Many downstream redistributors have policies against this kind of bundling, and
|
2016-01-19 12:57:17 +01:00
|
|
|
instead opt to patch the software they distribute to debundle it and make it
|
|
|
|
rely on the global versions of the software that they already have packaged
|
|
|
|
(which may have its own patches applied to it). We (the pip team) would prefer
|
|
|
|
it if pip was *not* debundled in this manner due to the above reasons and
|
|
|
|
instead we would prefer it if pip would be left intact as it is now. The one
|
|
|
|
exception to this, is it is acceptable to remove the
|
|
|
|
``pip/_vendor/requests/cacert.pem`` file provided you ensure that the
|
|
|
|
``ssl.get_default_verify_paths().cafile`` API returns the correct CA bundle for
|
|
|
|
your system. This will ensure that pip will use your system provided CA bundle
|
|
|
|
instead of the copy bundled with pip.
|
|
|
|
|
2016-01-19 14:52:21 +01:00
|
|
|
In the longer term, if someone has a *portable* solution to the above problems,
|
|
|
|
other than the bundling method we currently use, that doesn't add additional
|
|
|
|
problems that are unreasonable then we would be happy to consider, and possibly
|
|
|
|
switch to said method. This solution must function correctly across all of the
|
|
|
|
situation that we expect pip to be used and not mandate some external mechanism
|
|
|
|
such as OS packages.
|
2016-01-19 12:57:17 +01:00
|
|
|
|
|
|
|
|
2013-08-16 18:56:16 +02:00
|
|
|
Modifications
|
2016-01-19 12:57:17 +01:00
|
|
|
-------------
|
2013-08-16 18:56:16 +02:00
|
|
|
|
2016-11-23 16:37:19 +01:00
|
|
|
* ``setuptools`` is completely stripped to only keep ``pkg_resources``
|
|
|
|
* ``pkg_resources`` has been modified to import its dependencies from ``pip._vendor``
|
|
|
|
* ``packaging`` has been modified to import its dependencies from ``pip._vendor``
|
2017-12-16 13:37:49 +01:00
|
|
|
* ``html5lib`` has been modified to ``import six from pip._vendor``
|
|
|
|
* ``CacheControl`` has been modified to import its dependencies from ``pip._vendor``
|
|
|
|
* ``requests`` has been modified to import its other dependencies from ``pip._vendor``
|
|
|
|
and to *not* load ``simplejson`` (all platforms) and ``pyopenssl`` (Windows).
|
2014-01-07 10:40:32 +01:00
|
|
|
|
|
|
|
|
2016-11-06 23:34:10 +01:00
|
|
|
Automatic Vendoring
|
|
|
|
-------------------
|
|
|
|
|
|
|
|
Vendoring is automated via the ``vendoring.update`` task (defined in
|
|
|
|
``tasks/vendoring/__init__.py``) from the content of
|
|
|
|
``pip/_vendor/vendor.txt`` and the different patches in
|
|
|
|
``tasks/vendoring/patches/``.
|
2016-11-23 16:37:19 +01:00
|
|
|
Launch it via ``invoke vendoring.update`` (requires ``invoke>=0.13.0``).
|
2016-11-06 23:34:10 +01:00
|
|
|
|
|
|
|
|
2016-01-19 12:57:17 +01:00
|
|
|
Debundling
|
|
|
|
----------
|
2014-01-07 10:40:32 +01:00
|
|
|
|
2016-11-23 16:37:19 +01:00
|
|
|
As mentioned in the rationale, we, the pip team, would prefer it if pip was not
|
2016-01-19 12:57:17 +01:00
|
|
|
debundled (other than optionally ``pip/_vendor/requests/cacert.pem``) and that
|
2016-11-23 16:37:19 +01:00
|
|
|
pip was left intact. However, if you insist on doing so, we have a
|
2018-07-15 15:07:24 +02:00
|
|
|
semi-supported method (that we don't test in our CI) and requires a bit of
|
2016-11-23 16:37:19 +01:00
|
|
|
extra work on your end in order to solve the problems described above.
|
2013-08-16 18:56:16 +02:00
|
|
|
|
2016-01-19 12:57:17 +01:00
|
|
|
1. Delete everything in ``pip/_vendor/`` **except** for
|
|
|
|
``pip/_vendor/__init__.py``.
|
2013-08-16 18:56:16 +02:00
|
|
|
|
2016-01-19 12:57:17 +01:00
|
|
|
2. Generate wheels for each of pip's dependencies (and any of their
|
|
|
|
dependencies) using your patched copies of these libraries. These must be
|
2016-11-23 16:37:19 +01:00
|
|
|
placed somewhere on the filesystem that pip can access (``pip/_vendor`` is
|
|
|
|
the default assumption).
|
2013-08-16 18:56:16 +02:00
|
|
|
|
2016-01-19 12:57:17 +01:00
|
|
|
3. Modify ``pip/_vendor/__init__.py`` so that the ``DEBUNDLED`` variable is
|
|
|
|
``True``.
|
2013-08-16 18:56:16 +02:00
|
|
|
|
2018-07-15 15:08:44 +02:00
|
|
|
4. Upon installation, the ``INSTALLER`` file in pip's own ``dist-info``
|
|
|
|
directory should be set to something other than ``pip``, so that pip
|
|
|
|
can detect that it wasn't installed using itself.
|
|
|
|
|
|
|
|
5. *(optional)* If you've placed the wheels in a location other than
|
2016-11-23 16:37:19 +01:00
|
|
|
``pip/_vendor/``, then modify ``pip/_vendor/__init__.py`` so that the
|
2016-01-19 12:57:17 +01:00
|
|
|
``WHEEL_DIR`` variable points to the location you've placed them.
|
2018-07-15 15:08:44 +02:00
|
|
|
|
|
|
|
6. *(optional)* Update the ``pip_version_check`` logic to use the
|
|
|
|
appropriate logic for determining the latest available version of pip and
|
|
|
|
prompt the user with the correct upgrade message.
|