Implement Advanced Content Filter #97

Merged

pfm merged 36 commits from daemon into main

2022-10-19 20:48:41 +02:00

pfm commented

2022-10-17 20:25:22 +02:00

Collaborator

Goals:

Make code more reusable.
Improve expected performance by using asyncio.
Use requirements.txt file to document dependencies.

Goals: - Make code more reusable. - Improve expected performance by using asyncio. - Use requirements.txt file to document dependencies.

pfm added 34 commits 2022-10-17 20:25:24 +02:00

447da78c19 Extract a predicate to classify messages as already encrypted

Also: perform minor cleanup.

74003fca2d Fix logging

First initialise logging, then import lacre.mailgate module.  Otherwise,
module's logging quitely initialises its own root logger that doesn't use
configuration provided by the user.

Also: remove unnecessary "global" keywords.

7132b4c5b7 Restore keyring before running E2E tests

93126d200d Create skeleton of the Lacre daemon

Also:
- Expose a function to read mail relay configuration.
- Replace tabs with 4 spaces in lacre.config.

43e40a6e6d Mailgate: replace tabs with spaces

f8ec3f26dc Extend the daemon skeleton

a99ed42939 [daemon] Add configuration, implement no-op filter

- Add a "mailop" module to define mail operations.  Each should inherit from
MailOperation class (which just defines the contract).

- Make lacre.mailgate.delivery_plan always return KeepIntact strategy to have
a daemon that just forwards messages without modifying them.

- Add sample configuration.

- Include daemon configuration in mandatory parameter check.

ad94385f9b Implement a bare minimum of advanced filtering

- Forward messages without encryption.

- Include a simple test setup in the Makefile.

- Add a test to send a test message to the daemon.

a8bfe3bdaf Move different parts of Lacre tests to subdirectories

In particular, move:

- test utilities to test/utils;
- unit tests to test/modules.

Also: start implementing the Lacre daemon test (just a stub for now).

029e332d16 Implement E2E tests for lacre.daemon

- Add a dedicated configuration file for lacre.daemon.

- Implement test/daemon_test.py like test/e2e_test.py, to automate the
following procedure:

    1. Start up lacre.daemon.
    2. For each test case, send test message to the daemon and verify that the
       output received by test/utils/relay.py contains expected pattern.

- Simplify Makefile.

- Fix indentation here and there.

a5c525e45e Update documentation to cover daemon tests

5d8ad051c5 Split _gpg_encrypt into smaller functions

986844fa47 Continue splitting _sort_gpg_recipients

Extract new functions to match keys using enc_keymap and enc_domain_keymap
configuration sections, another one to look them up directly in GnuPG keyring,
optionally stripping delimiters ("+" followed by a topic).

Add some comments and docstrings.

b487170160 Add more debug logging to _try_direct_key_lookup

67d71ef1d5 Change indentation from tabs to 4 spaces

84b36eaa5c Fix a bug introduced by refactoring, clean up code

- Fix certificate retrieval.

- Store recipients within MailOperation objects.

- Log more information.

- Fix some warnings.

0f33b51d30 Refactor into smaller functions and objects

9f1b280962 Make daemon E2E tests use configured parameters

ae2b50fb94 Reformat GnuPG module

33c325a992 Continue refactoring

- Add more encryption strategies.
- Replace tuples (email + key) with dedicated objects.

e8b7a9fb44 Implement Advanced Filter flow for cleartext and OpenPGP

- Polish implementation of mail operations (lacre/mailop.py).  Add two
strategies: InlineOpenPGPEncrypt and MimeOpenPGPEncrypt, to support two modes
of OpenPGP encryption.

- In delivery_plan, only use those strategies that actually make sense with
the recipients we'd got.

- Add flag_enabled predicate (lacre/config.py) to make configuration checks
easier / simpler.

- Handle TypeError errors in Advanced Filter, indicating a delivery failure
when they appear.

- Add type hints to some of the functions.

8acf53c396 Reorder tests to avoid interferences

1c25959b3e Reformat tests

f52177bcb0 Implement a basic KeyCache

75a03de24b Add cache validity configuration parameter

Also, log basic information in KeyCache and provide load() and reload()
operations to make daemon's code cleaner.

119762977d Reload key cache periodically

Use [default]cache_refresh_minutes configuration parameter to define periods
between cache reloads.  After this number of minutes cache will be reloaded.

c583872250 Reload key cache only if keyring dir was modified

a3acd591f5 Log processing time for successful deliveries

cf01986464 Document dependencies on the logging module

b15d6f36f8 Reload keyring on filesystem events

Subscribe to FS events from keyring directory using Python Watchdog and when a
modification is observed, reload the key cache.

Since we may receive more than one event about a single modification, keep
directory's last modification to recognise 'false positives'.

97a6fee81b Reformat code, add doc comments

5e3c8abde5 Document dependencies

Add requirements.txt file with versions of dependencies known to work well.

bae25791e0 Rename lacre.keycache to lacre.keyring

This will better reflect the fact we're doing more than just caching.

c3d9220f0b Separate key-cache and key-loader

Extract key-loading code to a dedicated class KeyRing in lacre.keyring module.
KeyCache only keeps a static map of identities, making it safe to use in
asynchronous context (and race condition resistant).

pfm commented

2022-10-17 20:26:33 +02:00

Author

Collaborator

This change addresses #46.

pfm added 2 commits 2022-10-17 23:40:56 +02:00

f3ffe1da61 Make key-loading async, remove unused parameter

4b7c6c639e Add basic daemon documentation

pfm added the

DEVELOPMENT

label 2022-10-19 20:07:27 +02:00

pfm scheduled this pull request to auto merge when all checks succeed 2022-10-19 20:12:10 +02:00