EmanuelLoos/gpg-lacre
1
0
Fork 0
forked from Disroot/gpg-lacre
Code Issues Pull requests Projects Releases Packages Wiki Activity

Update #1

Merged
EmanuelLoos merged 211 commits from Disroot/gpg-lacre:main into main 2024-11-05 13:25:10 +01:00
Conversation 0 Commits 211 Files changed 72 +3956 -1663
EmanuelLoos commented 2024-11-05 13:21:42 +01:00
Owner
Copy link
No description provided.
EmanuelLoos added 211 commits 2024-11-05 13:21:44 +01:00 
It's possible to trick Lacre by sending PGP markers in message body, causing
it to classify that message being already encrypted.  This test case is used
to reproduce this scenario.
If a user mentions PGP markers inside their message, we should not classify it
as already encrypted.
Reviewed-on: Disroot/gpg-lacre#103
Reviewed-by: muppeth <muppeth@no-reply@disroot.org>
Just some small Markdown syntax corrections.
Reviewed-on: Disroot/gpg-lacre#107
Reviewed-on: Disroot/gpg-lacre#108
Reviewed-on: Disroot/gpg-lacre#112
Reviewed-by: pfm <pfm@no-reply@disroot.org>
- Cover basic GnuPG functionality with tests (confirm_key, public_keys).
- Add a test public key file.
- Fix resource leak by closing streams opened by Popen.
Reviewed-on: Disroot/gpg-lacre#114
Reviewed-on: Disroot/gpg-lacre#116
Reviewed-by: pfm <pfm@no-reply@disroot.org>
We no longer support decrypting emails, so:

- Remove parameters from exmaple configuration file.
- Remove relevant section from installation instructions (INSTALL.md).
Reviewed-on: Disroot/gpg-lacre#121
Function asyncio.run creates a new event loop each time it's called and
executes coroutine in that new loop.  However, we want all our coroutines to
be executed from the same event loop, so we acquire a loop when lacre.daemon
starts and then use it to execute them later.

See: Disroot/gpg-lacre#109
Expose a new parameter: [daemon]max_data_bytes, to limit Lacre's memory
usage and allow processing of messages larger than 32MB (which is the
default limit).
smtplib.SMTP expects ASCII-only message bodies when message body is provided
as a 'str'.  If we pass a 'bytes', we need to choose encoding earlier and we
do this by calling 'as_bytes' on messages with SMTP policy, which takes care
of formatting the body properly.

As a result, ISO-8859-x messages are converted to Quoted Printable and UTF-8
messages are Base64-encoded.

Testing this behaviour is tricky, because we use the same SMTP client to send
test data.  For this reason, test code has become a bit ugly, but it does
exactly what we need.
Add a new test message to verify Lacre's behaviour when processing
UTF-8 messages with text in two different scripts (latin-based and cyrillic).

Also: log Content-Transfer-Encoding when logging headers is enabled.
When GnuPG refuses to encrypt a message (e.g. when key has expired), record
information about the failure and send to logs, then deliver cleartext.  This
way we won't bounce email that could be delivered without encryption.

Also: add more E2E tests.
Also:
- Pass text to Popen in GnuPG (used to be bytes).
- Make is_payload_pgp_inline type-agnostic (str / bytes).
- Clean up PGP/MIME flow by using API instead of explicit/manual generation of
  headers.
- Fix E2E test configuration for PGP/MIME case.
- Add first lacre.core unit tests.
- Add another Contract Test.
- Use MIMEPart instead of Message when encrypting in PGP/MIME mode.

- Wrap text/plain messages in MIMEPart, instead of manipulating payloads
  manually.

- Add a test for wrapping.
In the daemon, specify policy as SMTPUTF8.  That sets the deafult message type
to EmailMessage.

EmailMessage class is richer, including support for Content Managers, giving
it the capability to properly handle textual data and its encodings.

Also: add another contract test.
Implement RecipientList class with:
- recipient (email) list,
- key (identity) list.

Cover with basic unit test.
Instead of passing pairs of lists (emails and keys) separately, implement a
class RecipientList to wrap such pair of lists.
Introduce modules:
- lacre.transport - for actual delivery via SMTP
- lacre.smime - to take care of S/MIME stuff

Implement lacre.transport.SendFrom class that does a almost exactly the same
thing as the original send_msg function, but without using global variable to
store original message sender.
We rely on Content Manager to select the right Content-Transfer-Encoding.
Also: adjust expected test output because it's now Base64-encoded.
- Remove unused code.
- Add docstrings.
Reviewed-on: Disroot/gpg-lacre#123
- ContentManager sets default Content-Type even if it was missing in the
  original message.

- Make sure that when Content-Type is missing, copying parameters doesn't
  raise an error.

- Add a unit-test to check that.
Reviewed-on: Disroot/gpg-lacre#125
- Simplify where possible.
- Remove outdated parts.
- Explain that only Python 3.9 is tested and supported.
- Link to Lacre Webgate repository.
- Explain requirements files are used now.
- Mention recommended Python version.
- Refresh instructions after splitting Lacre into components.
Reviewed-on: Disroot/gpg-lacre#128
When falling back to unencrypted mail delivery, do not pass sender information
to SendFrom.call method.
- Use regular expressions instead of finding particular characters in gnupg
  output to decide whether confirmation line was found.

- Use tempfile.mkdtemp to create secure temporary directories.

- Record information about the key considered by GnuPG. When missing in
  exception, it means no key was found.
Introduce new Python modules:

- lacre.notify -- to send notifications from the cron script;

- lacre.dbschema -- to keep database schema definition as code (SQLAlchemy);

- lacre.repositories -- to define key and identity repositories with high
  level APIs that we can then use elsewhere.

Also:

- rework GnuPG.add_key to return fingerprint so we can use it in the cron
  script;

- rename cron-job's logger name, replacing dash with an underscore as logging
  module doesn't like dashes.
Also: reformat with spaces instead of tabs.
Keys don't have to be surrounded with less-than and greater-than characters,
so this code could mishandle valid keys.
- Implement a context manager logging execution time.
- Use that context manager in daemon's handle_DATA method.
Also: rename key_id to fingerprint.
- Keep only one class to provide access to identities stored in the database.

- Remove old code and its tests.

- Align KeyRing and IdentityRepository APIs.

- Implement a (very) simple unit test for IdentityRepository.
- Flush key-collecting structures each time a new public key entry is found.
  This will avoid adding sub-keys and overwriting main keys with them.

- Use parseaddr from email.utils to parse emails (and drop realname part).

- Record logs produced during unit tests.

- Fix a small bug in test code.

Also: add basic information about available test identities to testing
documentation.
- Use accessor methods.
- Avoid data duplication.
- Don't pass table definitions to repository constructors.

- Keep an internal reference to Engine in lacre.repository.

- Implement KeyConfirmationQueue.count_keys.
- Let the user specify a directory, using the one from configuration by
  default.

- If user requested identity list without a specific email, list all.  Drop
  support for '-a' option.
Reviewed-on: Disroot/gpg-lacre#130
Reviewed-on: Disroot/gpg-lacre#133
With -r option, import command will first remove all identities and then load
them again from pubring.kbx.
Also: remove misleading comment about message.defects.
Reviewed-on: Disroot/gpg-lacre#134
- Introduce '[daemon]bounce_on_keys_missing' option to let the admin decide if
  they want Lacre to deliver cleartext message when identity database is
  unreachable or throws exceptions.  It defaults to 'no'.

- In IdentityRepository, use option mentioned above to decide what to do when
  an exception is caught.
Reviewed-on: Disroot/gpg-lacre#135
Provide 3 new configuration parameters in database section:

- max_connection_age --- number of seconds before an idle connection is
  "recycled", i.e. replaced with a new one;

- pool_size --- number of simultaneous connections kept in the pool;

- max_overflow --- maximum number of simultaneous connections we could make to
  the database.

Update sample config, including links to documentation.
- Provide a new reuqired parameter: [database]pooling_mode and use it during
  SQLAlchemy engine initialisation.

- Update tests and configuration (including sample configuration).

- Adjust repository unit test to load config during setup.

- Pass an engine instance to repository constructors instead of connections.
  Engine keeps a connection pool and we rely on it.
Also: use PGPStyle in lacre.core.
- Don't re-configure lacre.notify logger.
- Issue more DEBUG logs when deleting keys.
Reviewed-on: Disroot/gpg-lacre#136
Update naming in documentation and the source code.
SQLAlchemy's connection is a Context Manager and if we return a result from
code wrapped in a Context Manager, its cursor might already be closed.
- Set nullability of columns.
- Set up primary keys and auto-increment where necessary.
- Add missing 'lacre_locks' table.
- Implement a function to create tables.
- It supports option '-i' to initialise the schema.
- It logs a warning-level record of the schema manipulation.
Reviewed-on: Disroot/gpg-lacre#138
When we fail to produce byte representation of the email message being
processed, we may end up bouncing a message.  An example of such case would be
a message with a Message-Id header that Python's email parser library cannot
process.

In such cases, just take whatever original content we have received and pass
it to the destination without touching it to minimise any chances of breaking
the overall flow.
When we know we need to bounce a message and [daemon]log_headers is enabled,
we record up to 2.5kB of message headers at ERROR level.  This could help
diagnosing issues later.

Also: no longer record MIME Type, Charset and Content-Transfer-Encoding, as
the issues related to these properties no longer occur.
Reviewed-on: Disroot/gpg-lacre#144
Also: log more info when the daemon starts.
Reviewed-on: Disroot/gpg-lacre#145
Also: cover ExecutionTimeLogger with a unit test.
Reviewed-on: Disroot/gpg-lacre#146
1. Log the full traceback only once for each error (when we bounce the
message).

2. Use 451 response code on processing failure.

3. Disable decoding message contents as we operate on raw data anyway.
- Introduce exceptions to be raised upon transient and permanent delivery
failures, as specified by SMTP RFC.  Depending on type of failure, return
either 451 or 554 reply code.

- When serialising a message, treat ValueError as a serialisation issue (and
try again to deliver in cleartext).
- Implement KeyConfirmationQueue.delete_expired_queue_items to delete items
older than a given number of hours.

- Introduce configuration parameter to specify maximum number of hours.  It
defaults to 1 hour.

- Update documentation to explain that we never assign ST_TO_BE_DELETED.
- Always default to 'python' if PYTHON env. var. unset.
- Enable SQLAlchemy warnings in daemon tests.
- Commit changes in schema initialisation script.
We want to avoid deserialising message contents, because Python's email module
might produce different representation than the MUA sending original message.
The result would be a transformed message, which could mean broken message in
certain conditions.
Lacre no longer transforms cleartext payloads.
Reviewed-on: Disroot/gpg-lacre#152
Reviewed-on: Disroot/gpg-lacre#155
Reviewed-by: muppeth <muppeth@no-reply@disroot.org>
Reviewed-on: Disroot/gpg-lacre#157
Reviewed-on: Disroot/gpg-lacre#156
When we fail to encrypt a message, we log a message with a complete traceback.
The goal is to ensure that when diagnosing it later, we have complete
information.
Reviewed-on: Disroot/gpg-lacre#160
Reviewed-on: Disroot/gpg-lacre#162
Improves logging and some failure handling.

Reviewed-on: Disroot/gpg-lacre#163
EmanuelLoos merged commit a5e90f40c9 into main 2024-11-05 13:25:10 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: EmanuelLoos/gpg-lacre#1
No description provided.
Delete branch "Disroot/gpg-lacre:main"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?