proxpy
======

Dependence:
-----------

Both side:

- `Python`_ >= 3.8, built with ssl support.
- `PyNaCl`_

Client side:

- **openssl** executable, from `OpenSSL`_ >= 1.1 or `LibreSSL`_
- `brotli`_ or `brotlicffi`_ python module

  (optional, only used for decompressing cache data. compressed data will be cached if module not available)

- `zstd`_ or `zstandard`_ or `pyzstd`_ python module

  (optional, only used for decompressing cache data. compressed data will be cached if module not available)

Server side:

- `uWSGI`_ >= 2.0, built with python >= 3.8 support

.. _Python: https://www.python.org/
.. _PyNaCl: https://pynacl.readthedocs.io/
.. _OpenSSL: https://www.openssl.org/
.. _LibreSSL: https://www.libressl.org/
.. _brotli: https://github.com/google/brotli
.. _brotlicffi: https://github.com/python-hyper/brotlicffi
.. _zstd: https://github.com/sergey-dryabzhinsky/python-zstd
.. _zstandard: https://github.com/indygreg/python-zstandard
.. _pyzstd: https://github.com/animalize/pyzstd
.. _uWSGI: https://uwsgi-docs.readthedocs.io/
.. _Let’s Encrypt: https://letsencrypt.org/certificates/
.. _DigiCert: https://www.digicert.com/digicert-root-certificates.htm
.. _ca-certificates: https://packages.debian.org/stable/ca-certificates
.. _Cloudflare: https://developers.cloudflare.com/1.1.1.1/dns-over-https/
.. _Quad9: https://www.quad9.net/doh-quad9-dns-servers/
.. _Google Public DNS: https://developers.google.com/speed/public-dns/docs/doh/json
.. _Public recursive name server: https://en.wikipedia.org/wiki/Public_recursive_name_server
.. _HTTPS Everywhere Rulesets: https://www.eff.org/https-everywhere/rulesets

================== ======
File and directory  Used
================== ======
certfilter/         both
croxy/             client
crypto/             both
dohjson/            both
http0/              both
mkcert/            client
proxpycfg/          both
recert/            client
redirector/        client
streampacker/       both
cli_config.py      client
mkcertcli.py       unused
proxpy.py          client
remote.py          server
update_recerts.py  client
update_ruleset.py  client
uwsgi-sample.ini   server
================== ======

Usage:
------

- **important** choose and download CA certificate, e.g. `Let’s Encrypt`_ or `DigiCert`_
  (or unpack some certificates from `ca-certificates`_)

- **important** choose DNS provider, e.g. `Cloudflare`_, `Quad9`_ or `Google Public DNS`_
  (or carefully choose one from `Public recursive name server`_ that support DNS over HTTPS with JSON format)

- client side config overview:

::

   $>> python3 cli_config.py client --help
   usage: cli_config client [-h] MODE ...

   positional arguments:
     MODE
       create    create/overwrite with empty config.
       setdns    set DNS over HTTPS option.
       sethttp   set HTTP connection option.
       setcert   set proxy side certificate option.
       setproxy  add or set proxy (overwrite existing one).
       delproxy  delete proxy.
       remote    set remote server (overwrite existing one).
       show      show common config value.

   optional arguments:
     -h, --help  show this help message and exit

- create new client side config (prompt password if not provided in arguments):

::

   $>> python3 cli_config.py client create --help
   usage: cli_config client create [-h] [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

- set DNS over HTTPS in client side config (the CACERT is independent from others):

::

   $>> python3 cli_config.py client setdns --help
   usage: cli_config client setdns [-h] --ip IP [--port PORT] --name NAME [--path PATH] --cacert CACERT [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --ip IP              ip address of DNS over HTTPS server
     --port PORT          port of DNS over HTTPS server (default: 443)
     --name NAME          name of DNS over HTTPS server
     --path PATH          path of DNS over HTTPS server (default: /dns-query)
     --cacert CACERT      certificate used to verify DNS over HTTPS server (absolute path or relative path to <proxpy directory>/dohjson)
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

- set outgoing http options in client side config:

::

   $>> python3 cli_config.py client sethttp --help
   usage: cli_config client sethttp [-h] --cacert CACERT [--cache CACHE] [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --cacert CACERT      certificate used to verify https connection (absolute path or relative path to <proxpy directory>)
     --cache CACHE        path to store http cache (absolute path or relative path to <proxpy directory>/http0) (cache is always disabled in server side) (default: static)
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

- set certificate options of proxy server in client side config:

  (NOTE: supported DIGEST are varied with openssl executable, e.g. LibreSSL does not export sha3-* but OpenSSL does)

  (NOTE: the ed25519 key algorithm is not widely supported by browsers and not supported by LibreSSL)

  (NOTE: the secp521r1 curve is supported by Mozilla Firefox, but not supported by win32 system library (IE, Edge and Chrome))

::

   $>> python3 cli_config.py client setcert --help
   usage: cli_config client setcert [-h] [--keyalg CURVE] [--curve CURVE] [--md DIGEST] [--ca-name NAME] [--ca-expire DAYS] [--site-expire DAYS] [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --keyalg CURVE       algorithm of proxy side certificate key (choose from 'ec', 'ed25519') (default: ec)
     --curve CURVE        EC curve name of proxy side certificate key (choose from 'secp384r1', 'secp521r1') (default: secp521r1)
     --md DIGEST          Digest algorithm used proxy side certificate (choose from '...') (default: sha256)
     --ca-name NAME       commonName of proxy side certificate authority (default: Proxpy)
     --ca-expire DAYS     expire days of proxy side certificate authority (default: 30)
     --site-expire DAYS   expire days of proxy side per-site certificate. (default: 7)
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

- set proxy server in client side config:

::

   $>> python3 cli_config.py client setproxy --help
   usage: cli_config client setproxy [-h] --proxy-name NAME --proxy-type TYPE --proxy-ip IP --proxy-port PORT [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --proxy-name NAME    name of proxy to add/set
     --proxy-type TYPE    type of proxy to add/set (choose from 'cached', 'ccrypto', 'crypto', 'direct', 'nosni')
     --proxy-ip IP        bind ip of proxy to add/set
     --proxy-port PORT    bind port of proxy to add/set (range 1024 - 65535)
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

::

   proxy-type:
   - direct: direct access, nothing changed.
   - cached: direct access, cache something.
   - nosni: direct access without Server Name Indication (SNI)
   - cnosni: combination of nosni type with cached type
   - crypto: access through 'remote' (server side)
   - ccrypto: combination of crypto type with cached type

- delete proxy server from client side config:

::

   $>> python3 cli_config.py client delproxy --help
   usage: cli_config client delproxy [-h] --proxy-name NAME [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --proxy-name NAME    name of proxy to delete
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

- set remote server (server side) in client side config (need a working remote server, will require the public key and signature from remote server):

::

   $>> python3 cli_config.py client remote --help
   usage: cli_config client remote [-h] [--remote-ip IP] --remote-name NAME [--remote-port PORT] [--remote-path PATH] [--remote-nohttps] [--remote-nosni] [--cacert CACERT] [--cache CACHE] [--passwd PASSWD] [--loadpasswd PASSWD] [--chpasswd] [--nopasswd]

   optional arguments:
     -h, --help           show this help message and exit
     --remote-ip IP       remote ip (empty to auto resolve from --remote-name)
     --remote-name NAME   remote name
     --remote-port PORT   remote port (default according to protocol)
     --remote-path PATH   remote root path (default: /)
     --remote-nohttps     use http instead of https
     --remote-nosni       disable sni in https
     --cacert CACERT      certificate used to verify https connection (only used in config, default to system default)
     --cache CACHE        path to store http cache (absolute path or relative path to <proxpy directory>/croxy) (default: static)
     --passwd PASSWD      password to save config file (ignored without --chpasswd if different to --loadpasswd)
     --loadpasswd PASSWD  password to load config file (suppress prompt, default to use the same of --pass if provided)
     --chpasswd           change password of config file
     --nopasswd           disable password protection for config file (suppress prompt, --passwd and --chpasswd option)

- start client side:

::

   $>> python3 proxpy.py --help
   usage: cli_config [-h] [-P PASSWORD]

   optional arguments:
     -h, --help   show this help message and exit
     -P PASSWORD  password to load config file.

- server side config overview:

::

   $>> python3 cli_config.py server --help
   usage: cli_config server [-h] MODE ...

   positional arguments:
     MODE
       create    create/overwrite with empty config.
       setpath   set root path.
       setdns    set DNS over HTTPS option.
       sethttp   set HTTP connection option.
       show      show common config value.

   optional arguments:
     -h, --help  show this help message and exit

- set root path of server side

::

   $>> python3 cli_config.py server setpath --help
   usage: cli_config server setpath [-h] [--path PATH]

   optional arguments:
     -h, --help   show this help message and exit
     --path PATH  server root path (absolute path) (default: /)

- set DNS over HTTPS in server side config (the CACERT is independent from others):

::

   $>> python3 cli_config.py server setdns --help
   usage: cli_config server setdns [-h] --ip IP [--port PORT] --name NAME [--path PATH] --cacert CACERT

   optional arguments:
     -h, --help       show this help message and exit
     --ip IP          ip address of DNS over HTTPS server
     --port PORT      port of DNS over HTTPS server (default: 443)
     --name NAME      name of DNS over HTTPS server
     --path PATH      path of DNS over HTTPS server (default: /dns-query)
     --cacert CACERT  certificate used to verify DNS over HTTPS server (absolute path or relative path to <proxpy directory>/dohjson)

- set outgoing http options in server side config (cache is always disabled in server side):

::

   $>> python3 cli_config.py server sethttp --help
   usage: cli_config server sethttp [-h] --cacert CACERT [--cache CACHE]

   optional arguments:
     -h, --help       show this help message and exit
     --cacert CACERT  certificate used to verify https connection (absolute path or relative path to <proxpy directory>)
     --cache CACHE    path to store http cache (absolute path or relative path to <proxpy directory>/http0) (cache is always disabled in server side) (default: static)

- start server side

  ``PORT=<LISTEN PORT> uwsgi uwsgi.ini``

Advanced Usage:
---------------

- Hosts file

  - system hosts file is always ignored (`/etc/hosts` in linux, `C:\\Windows\\System32\\drivers\\etc\\hosts` in win32).

  - placed in <proxpy directory>/dohjson/hosts, ignored if not exists.

  - using the same format with the system hosts file.

  - effected once the `mtime` of file changed, suppress dns query if matched, pick the last result if multiple exist.

- Certificate filter

  - matched certificate will be refused.

  - filter rules placed <proxpy directory>/certfilter/rules, ignored if not exists.

  - effected once the `mtime` of file changed.

  - example of filter rules file:

::

  #-started lines will be ignored. empty lines will be also ignored.
  # there are three parts in earh line, seperated by one or more white space:
  #
  # <side> <field> <regular expression>
  #
  # no leading space is allowed.
  #
  # 'side' is case-sensitive and should be one of:
  #   'issuer', 'subject', 'subjectAltName'
  #
  # 'field' is case-insensitive and should be one of:
  #   'c', 'st', 'l', 'o', 'ou', 'cn'
  #
  # 'regular expression' will be used to test the value of certificate by re.search.
  # any invalid line will cause an exception.
  #
  # e.g.
  # to refuse any certificate issued by a common name that starts with 'WoTrus':
  issuer cn ^WoTrus.*$

- Redirector

  - **NOT COMPATIBLE** with the `HTTPS Everywhere Rulesets`_, since the regular expression of python and javascript are slightly different.

  - all files in `redirector/ruleset` end with `.xml` will be used as ruleset source.

  - run ``python3 update_ruleset.py`` after client configured to make the ruleset database.

  - new ruleset database effected immediately after ``python3 update_ruleset.py`` successfully called.

  - rules will be sorted by ruleset name, first match will be used.

  - if the rule is a 'explicit rule' (default), client will receive 308 code with the redirected location. (take care with loop redirect!)

  - if the rule is a 'implicit rule' (see example below), data will be received directly from redirected location and client will know nothing about redirection.

  - 'backend' changing only works with 'implicit rule'.

    'filtered' 'backend' will cause succeeded response with empty content.

    other invalid 'backend' name will cause 500 error.

    see 'proxy-type' in client side config for valid 'backend' name.

  - example of ruleset file:

::

  <ruleset name="RULESET_NAME"> <!-- duplicate ruleset name will cause error in update_ruleset, ruleset with 'default_off' attribute will be ignored no matter the value -->
    <target host="example.org"/> <!-- match full hostname -->
    <target host="*.another_example.org"/> <!-- match only one level wildcard in left -->
    <target host="*.com"/> <!-- no, top-level-only with wildcard is not supported and will be ignored -->
    <target host="1.2.3.4"/> <!-- ip address is also supported -->
    <target host="*.2.3.4"/> <!-- ip address matching with widecard is not supported, so 1.2.3.4 or 2.2.3.4 is not matched -->
    <exclusion pattern="exclude_pattern"/> <!-- all rules in ruleset will be ignored if matched in python: 're.match(pattern,url,flags=re.A|re.I)' -->
    <rule from="from_pattern" to="to_pattern"/> <!-- call in python: 're.sub(from_pattern,to_pattern,url,flags=re.A|re.I)' -->
    <rule from="from_pattern" to="to_pattern" implicit="anything or nothing"/> <!-- implicit rule only if 'implicit' attribute exists in rule, no matter the value -->
    <rule from="from_pattern" to="to_pattern" implicit="" backend="direct"/> <!-- change the backend used in implicit rule ('direct' in this example) -->
    <rule from="from_pattern" implicit="" backend="filtered"/> <!-- 'filtered' backend, response empty content for matched url -->
    <rule from="from_pattern" implicit="" backend="nosni"/> <!-- rule without 'to' attribute, url not changed. useful if only want to change backend -->
    <rule from="from_pattern" to="to_pattern" implicit="" backend="no, never use other invalid name in backend attribute"/>
    <rule from="from_pattern">
      <!-- header name should be in lower case -->
      <request header="request-header-name" value="send request-header-name header with this value to server, override existing value"/>
      <request header="remove-request-header-name"/> <!-- do not send 'remove-request-header-name' header to server -->
      <response header="response-header-name" value="client will receive response-header-name header with this value, override existing value"/>
      <response header="remove-response-header-name"/> <!-- client will not receive 'remove-response-header-name' header -->
    </rule>
  </ruleset>

- Recert

  - **DO NOT USE IT UNLESS YOU KNOW WHAT YOU ARE DOING**

  - override certificate and/or server name for each site.

  - apply on any client connection.

  - usually only useful with 'nosni' backend (since server may response wrong cerficate).

  - run ``python3 update_recerts.py`` to maintain rules. see ``python3 update_recerts.py --help`` for usage.