Commit Graph

131 Commits

Author SHA1 Message Date
Piotr F. Mieszkowski 5e108c189a Replace file-based identity store with a dedicated db table 2023-10-29 19:39:08 +01:00
Piotr F. Mieszkowski 02edb4cc96 Validate keyring type config parameter on daemon startup 2023-10-27 23:53:17 +02:00
Piotr F. Mieszkowski 3dd6913599 Initialise db connection lazily, use isolated asyncio test case 2023-10-23 22:44:53 +02:00
Piotr F. Mieszkowski e5339d264c Improve asyncio usage 2023-10-23 22:35:27 +02:00
Piotr F. Mieszkowski 41442e5b59 Add basic support for RDBMS-based keyring 2023-09-30 22:38:33 +02:00
Piotr F. Mieszkowski fccabc083c Fix unencrypted delivery arguments
When falling back to unencrypted mail delivery, do not pass sender information
to SendFrom.call method.
2023-09-21 20:21:01 +02:00
Piotr F. Mieszkowski a30b5e7577 Handle missing Content-Type properly
- ContentManager sets default Content-Type even if it was missing in the
  original message.

- Make sure that when Content-Type is missing, copying parameters doesn't
  raise an error.

- Add a unit-test to check that.
2023-05-19 20:30:00 +02:00
Piotr F. Mieszkowski 518b823b5c Fix simple filter: pass policy to as_bytes()
Also: adjust expected test output because it's now Base64-encoded.
2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 61cf50effe Fix MIME content sub-type handling for non-plain text messages 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 34e8b6a4eb Move the last key-related function to keyring module 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 285f5dbf18 Don't overwrite CTE
We rely on Content Manager to select the right Content-Transfer-Encoding.
2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski fdd11dba14 Log more information about FS events while reloading keys 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 682de14630 Split the code into smaller modules
Introduce modules:
- lacre.transport - for actual delivery via SMTP
- lacre.smime - to take care of S/MIME stuff

Implement lacre.transport.SendFrom class that does a almost exactly the same
thing as the original send_msg function, but without using global variable to
store original message sender.
2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski ff6e0bfbdd Move recipient-processing code to a dedicated module 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 5f5b374f84 Unify send_msg, add more type hints 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski f4e21217c2 [simple-filter] Retry delivery recoding text parts 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 1f1fe1dadb Add lots of log messages 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 8a42f3fea1 Improve error-handling for simple filter and test relay 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski a5f79c1ae7 Wrap recipient lists
Instead of passing pairs of lists (emails and keys) separately, implement a
class RecipientList to wrap such pair of lists.
2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski b2bd6a9926 Encapsulate recipient lists
Implement RecipientList class with:
- recipient (email) list,
- key (identity) list.

Cover with basic unit test.
2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski 603a88489e Polish the code 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski ffd5f08ad9 Make PGP message recognition more thorough 2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski d342f206de Handle messages as EmailMessage
In the daemon, specify policy as SMTPUTF8.  That sets the deafult message type
to EmailMessage.

EmailMessage class is richer, including support for Content Managers, giving
it the capability to properly handle textual data and its encodings.

Also: add another contract test.
2023-05-08 22:17:02 +02:00
Piotr F. Mieszkowski ea8b246538 Clean up PGP/MIME flow
- Use MIMEPart instead of Message when encrypting in PGP/MIME mode.

- Wrap text/plain messages in MIMEPart, instead of manipulating payloads
  manually.

- Add a test for wrapping.
2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 765637fd3a Fix logging initialisation by fixing import order 2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 1a3ce89ce5 Always encrypt decoded payload 2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 3c8b792203 Fix line-discarding bug 2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 56101b86c0 Clean up the code after refactor
- Clean up PGP/MIME flow by using API instead of explicit/manual generation of
  headers.
- Fix E2E test configuration for PGP/MIME case.
- Add first lacre.core unit tests.
- Add another Contract Test.
2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 27b07e672d Rework PGP/MIME flow 2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 5e408259c0 Start using Content Manager
Also:
- Pass text to Popen in GnuPG (used to be bytes).
- Make is_payload_pgp_inline type-agnostic (str / bytes).
2023-05-08 22:17:01 +02:00
Piotr F. Mieszkowski 94e22caf8e When encryption fails, revert to cleartext delivery
When GnuPG refuses to encrypt a message (e.g. when key has expired), record
information about the failure and send to logs, then deliver cleartext.  This
way we won't bounce email that could be delivered without encryption.

Also: add more E2E tests.
2023-05-08 22:16:41 +02:00
Piotr F. Mieszkowski 1cdca1d06d Log up to 2,5kB of message headers 2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski 3c1544e423 [daemon] Parse Envelope.original_content 2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski b242edf098 Add a more sophisticated UTF-8 test case
Add a new test message to verify Lacre's behaviour when processing
UTF-8 messages with text in two different scripts (latin-based and cyrillic).

Also: log Content-Transfer-Encoding when logging headers is enabled.
2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski b94123e83e Use SMTPUTF8 policy, add more debug logging 2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski 4da4019321 Log message defects and optionally some non-PII headers 2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski b6bd36a460 Use bytes instead of str to hold message bodies
smtplib.SMTP expects ASCII-only message bodies when message body is provided
as a 'str'.  If we pass a 'bytes', we need to choose encoding earlier and we
do this by calling 'as_bytes' on messages with SMTP policy, which takes care
of formatting the body properly.

As a result, ISO-8859-x messages are converted to Quoted Printable and UTF-8
messages are Base64-encoded.

Testing this behaviour is tricky, because we use the same SMTP client to send
test data.  For this reason, test code has become a bit ugly, but it does
exactly what we need.
2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski 5eb687f0cd Add config parameter to set DATA size limit
Expose a new parameter: [daemon]max_data_bytes, to limit Lacre's memory
usage and allow processing of messages larger than 32MB (which is the
default limit).
2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski f6bf86c533 Re-use the same Event Loop for each coroutine
Function asyncio.run creates a new event loop each time it's called and
executes coroutine in that new loop.  However, we want all our coroutines to
be executed from the same event loop, so we acquire a loop when lacre.daemon
starts and then use it to execute them later.

See: Disroot/gpg-lacre#109
2023-05-08 22:14:24 +02:00
Piotr F. Mieszkowski 18c790f986 Only reload on pubring.kbx file modifications 2022-10-23 13:51:42 +02:00
Piotr F. Mieszkowski 53378b516e Add a test for message with PGP inline markers that's not encrypted
If a user mentions PGP markers inside their message, we should not classify it
as already encrypted.
2022-10-22 21:29:59 +02:00
Piotr F. Mieszkowski fc85cdb841 Rework PGP-Inline verification/recognition 2022-10-22 19:58:16 +02:00
Piotr F. Mieszkowski 2ac26c09ce Simplify code, improve log entries, add comments 2022-10-22 11:23:17 +02:00
Piotr F. Mieszkowski 8f8d9dc1b6 Rename mailgate.py to core.py 2022-10-22 11:23:04 +02:00
Piotr F. Mieszkowski 99e939bb4e Remove too verbose debug logs, implement repr() for KeyCache 2022-10-20 22:27:34 +02:00
Piotr F. Mieszkowski 540ca2adf3 Improve logging
- Report processing time in milliseconds.
- Use module names in log messages instead of file-names without extensions.
2022-10-20 21:56:01 +02:00
Piotr F. Mieszkowski 641253b3ec Make key-loading async, remove unused parameter 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 9696b7e997 Separate key-cache and key-loader
Extract key-loading code to a dedicated class KeyRing in lacre.keyring module.
KeyCache only keeps a static map of identities, making it safe to use in
asynchronous context (and race condition resistant).
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 9f3ad49f14 Rename lacre.keycache to lacre.keyring
This will better reflect the fact we're doing more than just caching.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 2da97a5a9a Reformat code, add doc comments 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski eb0d5a1326 Reload keyring on filesystem events
Subscribe to FS events from keyring directory using Python Watchdog and when a
modification is observed, reload the key cache.

Since we may receive more than one event about a single modification, keep
directory's last modification to recognise 'false positives'.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 386c23f9f8 Document dependencies on the logging module 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 1db0a09fa5 Log processing time for successful deliveries 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski a85b7b7a43 Reload key cache only if keyring dir was modified 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski f5cff3292a Reload key cache periodically
Use [default]cache_refresh_minutes configuration parameter to define periods
between cache reloads.  After this number of minutes cache will be reloaded.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski d7e4947afd Add cache validity configuration parameter
Also, log basic information in KeyCache and provide load() and reload()
operations to make daemon's code cleaner.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 5f601fa50c Implement a basic KeyCache 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski a2eeaeee9d Implement Advanced Filter flow for cleartext and OpenPGP
- Polish implementation of mail operations (lacre/mailop.py).  Add two
strategies: InlineOpenPGPEncrypt and MimeOpenPGPEncrypt, to support two modes
of OpenPGP encryption.

- In delivery_plan, only use those strategies that actually make sense with
the recipients we'd got.

- Add flag_enabled predicate (lacre/config.py) to make configuration checks
easier / simpler.

- Handle TypeError errors in Advanced Filter, indicating a delivery failure
when they appear.

- Add type hints to some of the functions.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski ce6a0c5466 Continue refactoring
- Add more encryption strategies.
- Replace tuples (email + key) with dedicated objects.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski d01865d21c Refactor into smaller functions and objects 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski ddcef93abb Fix a bug introduced by refactoring, clean up code
- Fix certificate retrieval.

- Store recipients within MailOperation objects.

- Log more information.

- Fix some warnings.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski ce2e55e90c Change indentation from tabs to 4 spaces 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 0cb656f89d Add more debug logging to _try_direct_key_lookup 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 603710c41e Continue splitting _sort_gpg_recipients
Extract new functions to match keys using enc_keymap and enc_domain_keymap
configuration sections, another one to look them up directly in GnuPG keyring,
optionally stripping delimiters ("+" followed by a topic).

Add some comments and docstrings.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 68e4a452d2 Split _gpg_encrypt into smaller functions 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 4c844384e3 Implement a bare minimum of advanced filtering
- Forward messages without encryption.

- Include a simple test setup in the Makefile.

- Add a test to send a test message to the daemon.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 6455c1a280 [daemon] Add configuration, implement no-op filter
- Add a "mailop" module to define mail operations.  Each should inherit from
MailOperation class (which just defines the contract).

- Make lacre.mailgate.delivery_plan always return KeepIntact strategy to have
a daemon that just forwards messages without modifying them.

- Add sample configuration.

- Include daemon configuration in mandatory parameter check.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 7849c55d9f Extend the daemon skeleton 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 29b5b50901 Mailgate: replace tabs with spaces 2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 3f2760ba2d Create skeleton of the Lacre daemon
Also:
- Expose a function to read mail relay configuration.
- Replace tabs with 4 spaces in lacre.config.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski b198f0c4f4 Fix logging
First initialise logging, then import lacre.mailgate module.  Otherwise,
module's logging quitely initialises its own root logger that doesn't use
configuration provided by the user.

Also: remove unnecessary "global" keywords.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 9e998b54e9 Extract a predicate to classify messages as already encrypted
Also: perform minor cleanup.
2022-10-19 18:36:23 +00:00
Piotr F. Mieszkowski 5ffbbec5f0 Rename sort_recipients to deliver_message, remove unused imports 2022-06-11 21:05:27 +02:00
Piotr F. Mieszkowski b627fde510 Move gpg-mailgate.py logic to lacre.mailgate module
gpg-mailgate.py script keeps its role, but only needs to call code defined in
lacre.mailgate.
2022-06-11 21:00:42 +02:00
Piotr F. Mieszkowski c86c620668 Extract delimiter support, add unit tests
Also: fix recursive call to get_cert_for_email.
2022-06-07 22:14:32 +02:00
Piotr F. Mieszkowski 937046eb17 Use CRLF for line-endings
Mail RFCs use CRLF for line endings and it turns out things may break in
strange ways if only LF is used.
2022-06-02 19:56:32 +02:00
Piotr F. Mieszkowski 46be24670c Fix charset resolution in Content-Type parser 2022-06-01 23:44:41 +02:00
Piotr F. Mieszkowski d3b1717290 Extract PGP/INLINE checks, remove unnecessary byte-check 2022-06-01 23:00:05 +02:00
Piotr F. Mieszkowski 4c6fdc52ec Check mandatory config early, add tests
Also: extend failover logging configuration with file-based handler to make
sure that the user gets _some_ logs even if they do not configure Lacre at
all.
2022-05-31 22:09:10 +02:00
Piotr F. Mieszkowski 75ccfb0850 Use logging module
- Replace custom logging code with calls to logging module.
- Use logging.config to provide configuration parameters.

To make Lacre's logging more flexible, use fileConfig from logging.config to
set up all parameters.  If the configuration file is missing, use dictConfig
with hardcoded reasonable defaults.
2022-05-06 19:39:56 +02:00
Piotr F. Mieszkowski d90b50f7e7 Extract config, separate logging, split into smaller functions
- Move configuration-processing code to a separate module (lacre.config) and
  provide a simple API to access configuration parameters.
- Prepare to use builtin logging module to log diagnostic data.
- Rework the configuration-processing file to make it cleaner.
- Log additional information while processing configuration.
- Reorder functions.
2022-05-06 19:39:56 +02:00