Compare commits

...

104 Commits

Author SHA1 Message Date
pfm d3f1aa3a02 Merge pull request 'Improve error handling' (#146) from error-handling into main
Reviewed-on: #146
2024-03-03 08:35:03 +00:00
Piotr F. Mieszkowski e28864074c
Log exception and traceback when we fail-over to cleartext 2024-03-03 09:26:50 +01:00
Piotr F. Mieszkowski 0ec1bc3056
Set default values for non-nullable lacre_keys columns 2024-03-03 09:25:14 +01:00
Piotr F. Mieszkowski 110ea885f2
Deliver cleartext if Unicode encoding or message serialisation fail 2024-03-02 18:36:41 +01:00
Piotr F. Mieszkowski 3138864d32
Include exception in ExecutionTimeLogger log record
Also: cover ExecutionTimeLogger with a unit test.
2024-03-02 18:06:51 +01:00
pfm 1d8acc8eb8 Merge pull request '142_lacreadm' (#145) from 142_lacreadm into main
Reviewed-on: #145
2024-03-01 19:50:10 +00:00
Piotr F. Mieszkowski 8006b96df2 Rename lacreadm wrapper, mention it in documentation 2024-03-01 19:47:10 +00:00
Piotr F. Mieszkowski f80e4ecb9e Implement a very thin wrapper around 'python -m lacre.admin' 2024-03-01 19:47:10 +00:00
Piotr F. Mieszkowski 676ff47933 Don't ignore 'bin' directory 2024-03-01 19:47:10 +00:00
Piotr F. Mieszkowski f1c135850c lacre.admin: Report misconfiguration
Also: log more info when the daemon starts.
2024-03-01 19:47:10 +00:00
pfm ccfaa39501 Merge pull request 'Fix unencrypted delivery' (#144) from v0.2_unencrypted-delivery-fix into main
Reviewed-on: #144
2024-03-01 19:34:19 +00:00
Piotr F. Mieszkowski 7806d8c32a
Log message headers on a hard error
When we know we need to bounce a message and [daemon]log_headers is enabled,
we record up to 2.5kB of message headers at ERROR level.  This could help
diagnosing issues later.

Also: no longer record MIME Type, Charset and Content-Transfer-Encoding, as
the issues related to these properties no longer occur.
2024-03-01 20:28:51 +01:00
Piotr F. Mieszkowski 04ca103494
Fix unencrypted delivery in case of message generation failure
When we fail to produce byte representation of the email message being
processed, we may end up bouncing a message.  An example of such case would be
a message with a Message-Id header that Python's email parser library cannot
process.

In such cases, just take whatever original content we have received and pass
it to the destination without touching it to minimise any chances of breaking
the overall flow.
2024-03-01 20:14:09 +01:00
pfm d75ded751e Merge pull request 'Rename GPG-Mailgate to Lacre' (#138) from 81_rename-to-lacre into main
Reviewed-on: #138
2024-02-23 07:35:21 +00:00
Piotr F. Mieszkowski f601080e87 lacre.admin: Add more 'queue' documentation, clean up 2024-02-23 08:33:47 +01:00
Piotr F. Mieszkowski 80c25f6d2e lacre.admin: Document new sub-command 'database' 2024-02-23 08:25:52 +01:00
Piotr F. Mieszkowski aa2eb604d4 lacre.admin: Add a sub-command to manipulate database schema
- It supports option '-i' to initialise the schema.
- It logs a warning-level record of the schema manipulation.
2024-02-21 21:10:49 +01:00
Piotr F. Mieszkowski f7e6708949 Adjust lacre.dbschema to reflect original schema.sql
- Set nullability of columns.
- Set up primary keys and auto-increment where necessary.
- Add missing 'lacre_locks' table.
- Implement a function to create tables.
2024-02-21 21:10:49 +01:00
Piotr F. Mieszkowski be615df6e4 Split webgate-cron.py into small functions 2024-01-21 11:28:46 +01:00
Piotr F. Mieszkowski bfd3541b18 Retrieve data from db result before returning from Context Manager
SQLAlchemy's connection is a Context Manager and if we return a result from
code wrapped in a Context Manager, its cursor might already be closed.
2024-01-20 18:52:47 +01:00
Piotr F. Mieszkowski 8d2bf403a7 Add lacre.admin queue --list option, log query parameters 2024-01-16 20:33:23 +01:00
Piotr F. Mieszkowski 55a369df83 Add debug sqlalchemy logs (disabled by default) 2024-01-08 22:45:59 +01:00
Piotr F. Mieszkowski cd67b0934e Unify configuration requirements 2024-01-08 22:19:10 +01:00
Piotr F. Mieszkowski 276e0d0cd4 Use one config for cron and daemon tests 2024-01-07 21:52:52 +01:00
Piotr F. Mieszkowski bc2fc53416 Update gitignore: generated config, project name 2024-01-06 15:46:33 +01:00
Piotr F. Mieszkowski 260a3f3e9c Configure pooling for cron tests 2024-01-06 15:26:18 +01:00
Piotr F. Mieszkowski a943b50adb Update test configuration after renaming 2024-01-06 15:06:36 +01:00
Piotr F. Mieszkowski a98ff611ee Continue renaming: config files, tests, docs 2024-01-06 14:45:09 +01:00
Piotr F. Mieszkowski ad3a54fcd7 Rename GPG-Mailgate to Lacre
Update naming in documentation and the source code.
2024-01-06 14:34:54 +01:00
Piotr F. Mieszkowski 7208f66527 Improve simple filter structure 2024-01-05 22:21:20 +01:00
Piotr F. Mieszkowski a09fd67a59 Make keys unexpirable 2024-01-05 22:11:14 +01:00
pfm 748fd00957 Merge pull request 'lacre.repositories: Configure SQLAlchemy connection pooling' (#136) from connection-pooling into main
Reviewed-on: #136
2024-01-04 18:52:03 +00:00
Piotr F. Mieszkowski 8f8f081d28 Fix key-removal condition, improve logging 2024-01-04 19:45:25 +01:00
Piotr F. Mieszkowski 07539a97d3 Improve logging
- Don't re-configure lacre.notify logger.
- Issue more DEBUG logs when deleting keys.
2023-12-20 23:03:04 +01:00
Piotr F. Mieszkowski 5c327b166a webgate-cron: Log more information, including exceptions 2023-12-19 18:21:00 +01:00
Piotr F. Mieszkowski 41b7535412 Add more logging, add --delete option to admin queue sub-command 2023-12-19 09:02:42 +01:00
Piotr F. Mieszkowski 9b5d578985 lacre.config: Make both enums case-insensitive
Also: use PGPStyle in lacre.core.
2023-12-17 20:42:57 +01:00
Piotr F. Mieszkowski ff429c93e6 Convert pooling parameters to integers 2023-12-17 14:12:52 +01:00
Piotr F. Mieszkowski 90da933bf9 Make disconnect handling configuration explicit
- Provide a new reuqired parameter: [database]pooling_mode and use it during
  SQLAlchemy engine initialisation.

- Update tests and configuration (including sample configuration).

- Adjust repository unit test to load config during setup.

- Pass an engine instance to repository constructors instead of connections.
  Engine keeps a connection pool and we rely on it.
2023-12-17 14:03:20 +01:00
Piotr F. Mieszkowski 86cc27e918 lacre.repositories: Configure SQLAlchemy connection pooling
Provide 3 new configuration parameters in database section:

- max_connection_age --- number of seconds before an idle connection is
  "recycled", i.e. replaced with a new one;

- pool_size --- number of simultaneous connections kept in the pool;

- max_overflow --- maximum number of simultaneous connections we could make to
  the database.

Update sample config, including links to documentation.
2023-12-16 23:32:27 +01:00
pfm 18a64bcd72 Merge pull request 'Add ability to deliver cleartext when keys can't be loaded' (#135) from fix/keys-not-loaded into main
Reviewed-on: #135
2023-12-10 20:41:48 +00:00
Piotr F. Mieszkowski e8d0d248b3 lacre.repositories: Add missing import 2023-12-10 21:39:59 +01:00
Piotr F. Mieszkowski 23a05c11ac Remove EncryptionException formatting test 2023-12-10 21:35:35 +01:00
Piotr F. Mieszkowski 8cc1136a90 lacre.daemon: When keys can't be loaded, fail gracefully
- Introduce '[daemon]bounce_on_keys_missing' option to let the admin decide if
  they want Lacre to deliver cleartext message when identity database is
  unreachable or throws exceptions.  It defaults to 'no'.

- In IdentityRepository, use option mentioned above to decide what to do when
  an exception is caught.
2023-12-10 21:27:05 +01:00
pfm 628de8a28d Merge pull request 'Fix cron script and more' (#134) from fix/cron-script into main
Reviewed-on: #134
2023-12-09 20:26:37 +00:00
Piotr F. Mieszkowski c0b98649d4 lacre.admin: Document import command 2023-12-09 21:14:35 +01:00
Piotr F. Mieszkowski fe2c0cbf76 Fix unprintable exception issue 2023-12-09 20:57:09 +01:00
Piotr F. Mieszkowski 75c48282b0 Rework encryption exception handling
Also: remove misleading comment about message.defects.
2023-12-09 20:38:46 +01:00
Piotr F. Mieszkowski fc08813bdc Improve unencryptable message logs 2023-12-09 19:48:20 +01:00
Piotr F. Mieszkowski d51c675881 lacre.admin: Make import -r option a flag (Boolean) 2023-12-05 21:51:35 +01:00
Piotr F. Mieszkowski abaf8820d7 lacre.admin: Add -r / --reload option to import command
With -r option, import command will first remove all identities and then load
them again from pubring.kbx.
2023-12-05 21:49:23 +01:00
Piotr F. Mieszkowski 94d0a62766 Identity removal: execute prepared DELETE 2023-12-05 21:33:19 +01:00
Piotr F. Mieszkowski cc1bacbe3d Move some imports to lacre.notify 2023-12-05 21:13:02 +01:00
pfm 4c603839b5 Merge pull request 'lacre.repositories: Fix IdentityRepository existence predicate' (#133) from 132-fix-repo-upsert into main
Reviewed-on: #133
2023-12-04 21:59:00 +00:00
Piotr F. Mieszkowski 0d852bc279 lacre.repositories: Fix IdentityRepository existence predicate 2023-12-04 22:57:43 +01:00
pfm b7713207ab Merge pull request 'Fix unencrypted delivery and key removal' (#130) from 129-key-removal into main
Reviewed-on: #130
2023-12-02 20:59:13 +00:00
Piotr F. Mieszkowski ac5dddfa98 Remove configuration options no longer used 2023-12-02 21:48:17 +01:00
Piotr F. Mieszkowski 052551072e Change table prefix from 'gpgmw' to 'lacre' 2023-12-02 20:02:59 +01:00
Piotr F. Mieszkowski 0975ce3a69 lacre.admin: Handle database exceptions 2023-11-26 19:52:58 +01:00
Piotr F. Mieszkowski b44bd7b150 lacre.admin: Implement identity import, fix identity list
- Let the user specify a directory, using the one from configuration by
  default.

- If user requested identity list without a specific email, list all.  Drop
  support for '-a' option.
2023-11-26 18:30:25 +01:00
Piotr F. Mieszkowski 0fe5e6b3dc Make GnuPG.public_keys docstring more complete 2023-11-26 18:29:43 +01:00
Piotr F. Mieszkowski aa8c353a05 Replace NBSP in doc/admin.md with a regular space 2023-11-26 18:29:09 +01:00
Piotr F. Mieszkowski 97c4f9f14a lacre.repositories: Fix naming after refactoring 2023-11-25 16:09:23 +01:00
Piotr F. Mieszkowski 626fce5f2c lacre.admin: Implement 'identities' sub-command 2023-11-25 16:08:54 +01:00
Piotr F. Mieszkowski 95c5802c38 Add test/lacre.db to gitignore 2023-11-25 15:09:00 +01:00
Piotr F. Mieszkowski 9b5c43b769 Fix crontest config, polish Makefile 2023-11-25 15:05:27 +01:00
Piotr F. Mieszkowski 7fe52ae8b5 Don't pass table definition to KeyConfirmationQueue 2023-11-25 15:02:48 +01:00
Piotr F. Mieszkowski 1ad0d2df0e Implement lacre.admin CLI tool 2023-11-25 14:07:10 +01:00
Piotr F. Mieszkowski becb39f139 Clean up database access
- Don't pass table definitions to repository constructors.

- Keep an internal reference to Engine in lacre.repository.

- Implement KeyConfirmationQueue.count_keys.
2023-11-25 14:04:32 +01:00
Piotr F. Mieszkowski 4950e0b9c3 Keep secondary keyring for test purposes 2023-11-25 01:13:43 +01:00
Piotr F. Mieszkowski acd33fec1e Fix inheritance issues
- Use accessor methods.
- Avoid data duplication.
2023-11-25 01:11:44 +01:00
Piotr F. Mieszkowski 72217e38ea GnuPG module: make key-listing more thorough
- Flush key-collecting structures each time a new public key entry is found.
  This will avoid adding sub-keys and overwriting main keys with them.

- Use parseaddr from email.utils to parse emails (and drop realname part).

- Record logs produced during unit tests.

- Fix a small bug in test code.

Also: add basic information about available test identities to testing
documentation.
2023-11-25 01:08:15 +01:00
Piotr F. Mieszkowski 7c2d32bf3c Make IdentityRepository a KeyRing
- Keep only one class to provide access to identities stored in the database.

- Remove old code and its tests.

- Align KeyRing and IdentityRepository APIs.

- Implement a (very) simple unit test for IdentityRepository.
2023-11-24 22:59:21 +01:00
Piotr F. Mieszkowski 5efef3c9cb Fix table name, unify metadata handling 2023-11-20 22:27:35 +01:00
Piotr F. Mieszkowski 89affde0d5 Add tests for GnuPG parsing routines 2023-11-20 22:11:37 +01:00
Piotr F. Mieszkowski bfa2643dc7 Implement identity repository
Also: rename key_id to fingerprint.
2023-11-20 22:11:25 +01:00
Piotr F. Mieszkowski 56da7e0cb4 Refactor calculating execution time
- Implement a context manager logging execution time.
- Use that context manager in daemon's handle_DATA method.
2023-11-20 22:03:59 +01:00
Piotr F. Mieszkowski 4fbae908d6 Don't require less-than and greater-than around the email
Keys don't have to be surrounded with less-than and greater-than characters,
so this code could mishandle valid keys.
2023-11-19 22:45:08 +01:00
Piotr F. Mieszkowski c6b2dbf618 Add docs, improve logging 2023-11-17 22:55:37 +01:00
Piotr F. Mieszkowski 7ac928af76 Handle gpg-mailgate.py missing params better 2023-11-17 22:51:09 +01:00
Piotr F. Mieszkowski a3eb892df9 Remove duplicate logger initialisation 2023-11-15 20:25:42 +01:00
Piotr F. Mieszkowski 2edd842f90 Use lacre.dbschema definition of identities table 2023-11-12 20:20:38 +01:00
Piotr F. Mieszkowski 6ca5db2db3 Issue an INFO log entry after configuring logging
Also: reformat with spaces instead of tabs.
2023-11-12 19:57:12 +01:00
Piotr F. Mieszkowski 9bbc86bc53 Extract parts of cron script to modules
Introduce new Python modules:

- lacre.notify -- to send notifications from the cron script;

- lacre.dbschema -- to keep database schema definition as code (SQLAlchemy);

- lacre.repositories -- to define key and identity repositories with high
  level APIs that we can then use elsewhere.

Also:

- rework GnuPG.add_key to return fingerprint so we can use it in the cron
  script;

- rename cron-job's logger name, replacing dash with an underscore as logging
  module doesn't like dashes.
2023-11-12 19:56:45 +01:00
Piotr F. Mieszkowski bf677585be Don't require watchdog anymore 2023-11-01 21:26:42 +01:00
Piotr F. Mieszkowski 5e108c189a Replace file-based identity store with a dedicated db table 2023-10-29 19:39:08 +01:00
Piotr F. Mieszkowski 02edb4cc96 Validate keyring type config parameter on daemon startup 2023-10-27 23:53:17 +02:00
Piotr F. Mieszkowski 3dd6913599 Initialise db connection lazily, use isolated asyncio test case 2023-10-23 22:44:53 +02:00
Piotr F. Mieszkowski e5339d264c Improve asyncio usage 2023-10-23 22:35:27 +02:00
Piotr F. Mieszkowski 43f43a4137 Fix DatabaseKeyring tests 2023-10-23 20:26:23 +02:00
Piotr F. Mieszkowski 41442e5b59 Add basic support for RDBMS-based keyring 2023-09-30 22:38:33 +02:00
Piotr F. Mieszkowski 274bfbaf3b Always use 'python' binary during tests 2023-09-30 22:33:49 +02:00
Piotr F. Mieszkowski c570bcd383 Update Alice's key expiry date 2023-09-25 19:44:37 +02:00
Piotr F. Mieszkowski 624a335a41 GnuPG: clean up and collect more diagnostic info
- Use regular expressions instead of finding particular characters in gnupg
  output to decide whether confirmation line was found.

- Use tempfile.mkdtemp to create secure temporary directories.

- Record information about the key considered by GnuPG. When missing in
  exception, it means no key was found.
2023-09-21 20:21:01 +02:00
Piotr F. Mieszkowski 6c114b6dcd Ensure correct logging initialisation in webgate-cron 2023-09-21 20:21:01 +02:00
Piotr F. Mieszkowski fccabc083c Fix unencrypted delivery arguments
When falling back to unencrypted mail delivery, do not pass sender information
to SendFrom.call method.
2023-09-21 20:21:01 +02:00
pfm 401f67844a Merge pull request 'Refresh docs' (#128) from 126-refresh-docs into main
Reviewed-on: #128
2023-07-08 13:40:42 +00:00
Piotr F. Mieszkowski cfbb413e7e Explicitly mention requirements.txt file 2023-07-08 02:03:20 +02:00
Piotr F. Mieszkowski adcafb30c3 Reorder and simplify first secions of README 2023-07-08 02:02:47 +02:00
Piotr F. Mieszkowski f0d4447f4a Move requirements to INSTALL, improve language 2023-06-25 22:41:13 +02:00
Piotr F. Mieszkowski addb119b3e Update INSTALL.md
- Explain requirements files are used now.
- Mention recommended Python version.
- Refresh instructions after splitting Lacre into components.
2023-06-25 22:31:34 +02:00
Piotr F. Mieszkowski bcd0284eac Update README
- Simplify where possible.
- Remove outdated parts.
- Explain that only Python 3.9 is tested and supported.
- Link to Lacre Webgate repository.
2023-06-18 21:31:47 +02:00
pfm c8f6743768 Merge pull request 'Handle missing Content-Type properly' (#125) from 124-missing-ct into main
Reviewed-on: #125
2023-05-19 18:38:17 +00:00
Piotr F. Mieszkowski a30b5e7577 Handle missing Content-Type properly
- ContentManager sets default Content-Type even if it was missing in the
  original message.

- Make sure that when Content-Type is missing, copying parameters doesn't
  raise an error.

- Add a unit-test to check that.
2023-05-19 20:30:00 +02:00
45 changed files with 1524 additions and 698 deletions

8
.gitignore vendored
View File

@ -1,3 +1,6 @@
# Generated project files:
test/lacre.db
*.py[cod]
# C extensions
@ -10,7 +13,6 @@ dist
build
eggs
parts
bin
var
sdist
develop-eggs
@ -26,10 +28,10 @@ pip-log.txt
.tox
nosetests.xml
# GPG-Mailgate test files
# Lacre test files
test/logs
test/tmp
test/gpg-mailgate.conf
test/lacre.conf
test/keyhome/random_seed
# Emacs files

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
"""GnuPG wrapper module."""
@ -27,28 +27,33 @@ import random
import string
import sys
import logging
import re
import tempfile
from email.utils import parseaddr
LINE_FINGERPRINT = 'fpr'
LINE_USER_ID = 'uid'
LINE_PUBLIC_KEY = 'pub'
POS_FINGERPRINT = 9
POS_UID = 9
LOG = logging.getLogger(__name__)
RX_CONFIRM = re.compile(br'key "([^"]+)" imported')
class EncryptionException(Exception):
"""Represents a failure to encrypt a payload."""
"""Represents a failure to encrypt a payload.
def __init__(self, issue: str, recipient: str, cause: str):
"""Initialise an exception."""
self._issue = issue
self._recipient = recipient
self._cause = cause
def __str__(self):
"""Return human-readable string representation."""
return f"issue: {self._issue}; to: {self._recipient}; cause: {self._cause}"
Arguments passed to exception constructor:
- issue: human-readable explanation of the issue;
- recipient: owner of the key;
- cause: any additional information, if present;
- key: fingerprint of the key.
"""
pass
def _build_command(key_home, *args, **kwargs):
@ -57,25 +62,40 @@ def _build_command(key_home, *args, **kwargs):
return cmd
def public_keys(keyhome):
"""List public keys from keyring KEYHOME."""
def public_keys(keyhome, *, key_id=None):
"""List public keys from keyring KEYHOME.
Returns a dict with fingerprints as keys and email as values."""
cmd = _build_command(keyhome, '--list-keys', '--with-colons')
if key_id is not None:
cmd.append(key_id)
p = subprocess.Popen(cmd, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.wait()
keys = dict()
collected = set()
fingerprint = None
email = None
for line in p.stdout.readlines():
line = line.decode(sys.getdefaultencoding())
if line[0:3] == LINE_FINGERPRINT:
fingerprint = line.split(':')[POS_FINGERPRINT]
if line[0:3] == LINE_PUBLIC_KEY:
# New identity has started, reset state.
fingerprint = None
email = None
if line[0:3] == LINE_FINGERPRINT and not fingerprint:
fingerprint = _extract_fingerprint(line)
if line[0:3] == LINE_USER_ID:
if ('<' not in line or '>' not in line):
continue
email = line.split('<')[1].split('>')[0]
if not (fingerprint is None or email is None):
email = _parse_uid_line(line)
if fingerprint and email and not email in collected:
keys[fingerprint] = email
collected.add(email)
fingerprint = None
email = None
@ -84,6 +104,23 @@ def public_keys(keyhome):
return keys
def _extract_fingerprint(line):
fpr_line = line.split(':')
if len(fpr_line) <= POS_FINGERPRINT:
return None
else:
return fpr_line[POS_FINGERPRINT]
def _parse_uid_line(line: str):
userid_line = line.split(':')
if len(userid_line) <= POS_UID:
return None
else:
(_, email) = parseaddr(userid_line[POS_UID])
return email
def _to_bytes(s) -> bytes:
if isinstance(s, str):
return bytes(s, sys.getdefaultencoding())
@ -94,32 +131,24 @@ def _to_bytes(s) -> bytes:
# Confirms a key has a given email address by importing it into a temporary
# keyring. If this operation succeeds and produces a message mentioning the
# expected email, a key is confirmed.
def confirm_key(content, email):
def confirm_key(content, email: str):
"""Verify that the key CONTENT is assigned to identity EMAIL."""
tmpkeyhome = ''
content = _to_bytes(content)
expected_email = _to_bytes(email.lower())
expected_email = email.lower()
while True:
tmpkeyhome = '/tmp/' + ''.join(random.choice(string.ascii_uppercase + string.digits) for x in range(12))
if not os.path.exists(tmpkeyhome):
break
tmpkeyhome = tempfile.mkdtemp()
LOG.debug('Importing into temporary directory: %s', tmpkeyhome)
# let only the owner access the directory, otherwise gpg would complain
os.mkdir(tmpkeyhome, mode=0o700)
localized_env = os.environ.copy()
localized_env["LANG"] = "C"
p = subprocess.Popen(_build_command(tmpkeyhome, '--import', '--batch'), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=localized_env)
result = p.communicate(input=content)[1]
result = _import_key(tmpkeyhome, content)
confirmed = False
for line in result.split(b"\n"):
if b'imported' in line and b'<' in line and b'>' in line:
if line.split(b'<')[1].split(b'>')[0].lower() == expected_email:
confirmed = True
break
else:
break # confirmation failed
for line in result.splitlines():
LOG.debug('Line from GnuPG: %s', line)
found = RX_CONFIRM.search(line)
if found:
(_, extracted_email) = parseaddr(found.group(1).decode())
confirmed = (extracted_email == expected_email)
LOG.debug('Confirmed email %s: %s', extracted_email, confirmed)
# cleanup
shutil.rmtree(tmpkeyhome)
@ -127,19 +156,44 @@ def confirm_key(content, email):
return confirmed
def _import_key(keyhome, content):
content = _to_bytes(content)
# Ensure we get expected output regardless of the system locale.
localized_env = os.environ.copy()
localized_env["LANG"] = "C"
p = subprocess.Popen(_build_command(keyhome, '--import', '--batch'), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=localized_env)
output = p.communicate(input=content)[1]
p.wait()
return output
# adds a key and ensures it has the given email address
def add_key(keyhome, content):
"""Register new key CONTENT in the keyring KEYHOME."""
if isinstance(content, str):
content = bytes(content, sys.getdefaultencoding())
p = subprocess.Popen(_build_command(keyhome, '--import', '--batch'), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate(input=content)
p.wait()
output = _import_key(keyhome, content)
email = None
for line in output.splitlines():
found = RX_CONFIRM.search(line)
if found:
(_, extracted_email) = parseaddr(found.group(1).decode())
email = extracted_email
# Find imported key to get its fingerprint
imported = public_keys(keyhome, key_id=email)
if len(imported.keys()) == 1:
fingerprint = list(imported.keys())[0]
return fingerprint, imported[fingerprint]
else:
return None, None
def delete_key(keyhome, email):
"""Remove key assigned to identity EMAIL from keyring KEYHOME."""
from email.utils import parseaddr
result = parseaddr(email)
if result[1]:
@ -149,6 +203,7 @@ def delete_key(keyhome, email):
p.wait()
return True
LOG.warn('Failed to parse email before deleting key: %s', email)
return False
@ -178,7 +233,7 @@ class GPGEncryptor:
if p.returncode != 0:
LOG.debug('Errors: %s', err)
details = parse_status(err)
raise EncryptionException(details['issue'], details['recipient'], details['cause'])
raise EncryptionException(details['issue'], details['recipient'], details['cause'], details['key'])
return (encdata, p.returncode)
def _popen(self):
@ -243,6 +298,8 @@ KEY_EXPIRED = b'KEYEXPIRED'
KEY_REVOKED = b'KEYREVOKED'
NO_RECIPIENTS = b'NO_RECP'
INVALID_RECIPIENT = b'INV_RECP'
KEY_CONSIDERED = b'KEY_CONSIDERED'
NOAVAIL = b'n/a'
# INV_RECP reason code descriptions.
INVALID_RECIPIENT_CAUSES = [
@ -271,7 +328,7 @@ def parse_status(status_buffer: str) -> dict:
def parse_status_lines(lines: list) -> dict:
"""Parse --status-fd output and return important information."""
result = {'issue': 'n/a', 'recipient': 'n/a', 'cause': 'Unknown'}
result = {'issue': NOAVAIL, 'recipient': NOAVAIL, 'cause': 'Unknown', 'key': NOAVAIL}
LOG.debug('Processing stderr lines %s', lines)
@ -281,15 +338,19 @@ def parse_status_lines(lines: list) -> dict:
continue
if line.startswith(KEY_EXPIRED, STATUS_FD_PREFIX_LEN):
result['issue'] = KEY_EXPIRED
result['issue'] = 'key expired'
elif line.startswith(KEY_REVOKED, STATUS_FD_PREFIX_LEN):
result['issue'] = KEY_REVOKED
result['issue'] = 'key revoked'
elif line.startswith(NO_RECIPIENTS, STATUS_FD_PREFIX_LEN):
result['issue'] = NO_RECIPIENTS
result['issue'] = 'no recipients'
elif line.startswith(KEY_CONSIDERED, STATUS_FD_PREFIX_LEN):
result['key'] = line.split(b' ')[2]
elif line.startswith(INVALID_RECIPIENT, STATUS_FD_PREFIX_LEN):
words = line.split(b' ')
reason_code = int(words[2])
result['recipient'] = words[3]
result['cause'] = INVALID_RECIPIENT_CAUSES[reason_code]
if reason_code:
result['cause'] = INVALID_RECIPIENT_CAUSES[reason_code]
return result

View File

@ -3,32 +3,33 @@
## Content
- General information
- Install GPG-Mailgate
- Install GPG-Mailgate-Web
- Install Lacre
- Install [Lacre-Webgate](https://git.disroot.org/Lacre/lacre-webgate)
- Install Register-handler
## General information
GPG-Mailgate is divided in 3 main parts: GPG-Mailgate itself, GPG-Mailgate-Web and Register-handler. Some parts of the GPG-Mailgate project depend on other parts of the project. You will find information about these dependencies at the beginning of every installation part.
Lacre is divided in 3 main parts: Lacre itself, Lacre-Webgate and Register-handler. Some parts of the Lacre project depend on other parts of the project. You will find information about these dependencies at the beginning of every installation part.
These instructions show you how to set up GPG-Mailgate in an easy way. If you are a more advanced user, feel free to experiment with the settings. For these instructions a home directory for the user `nobody` is set. Sadly this is an odd workaround but no better solution was found.
These instructions show you how to set up Lacre in an easy way. If you are a more advanced user, feel free to experiment with the settings. For these instructions a home directory for the user `nobody` is set. Sadly this is an odd workaround but no better solution was found.
These instructions are based on an installation on an Ubuntu 14.04 LTS virtual machine. For other Linux distributions and other versions these instructions might need to be adapted to your distribution (e.g. installation of packages and used directories).
## Install GPG-Mailgate
## Install Lacre
### Requirements
- Python 3.x is already installed
- Postfix is already installed and configured. It is recommended that you have already tested your configuration so we can exclude this as a main cause of problems
- GnuPG is already installed and configured
- Python 3.9.
- Dependencies listed in [requirements file](https://packaging.python.org/en/latest/tutorials/installing-packages/#requirements-files), `requirements.txt`.
- Postfix: installed, configured and tested.
- GnuPG: installed, configured and tested (e.g. via command-line).
### Installation
1. Install the Python-M2Crypto module:
1. Install the dependencies:
```
apt-get install python-m2crypto
python -m pip install -r requirements.txt
```
2. Set the home directory for the user `nobody` (sadly this workaround is needed as there is no better solution at this point). If you get an error that the user is currently used by a process, you might need to kill the process manually.
@ -40,29 +41,26 @@ usermod -d /var/gpgmailgate nobody
3. Create dedicated directories for storing PGP keys and S/MIME certificates and make the user `nobody` owner of these:
```
mkdir -p /var/gpgmailgate/.gnupg
mkdir -p /var/gpgmailgate/smime
chown -R nobody:nogroup /var/gpgmailgate/
install -u nobody -g nobody -d /var/gpgmailgate/ /var/gpgmailgate/.gnupg /var/gpgmailgate/smime
```
4. Place the `gpg-mailgate.py` in `/usr/local/bin/`, make the user `nobody` owner of the file and make it executable:
4. Place the `lacre.py` in `/usr/local/bin/`, make the user `nobody` owner of the file and make it executable:
```
chown nobody:nogroup /usr/local/bin/gpg-mailgate.py
chmod u+x /usr/local/bin/gpg-mailgate.py
install -u nobody -g nobody -mode u=rx lacre.py /usr/local/bin/
```
5. Place the `GnuPG` directory in `/usr/local/lib/python3.x/dist-packages` (replace 3.x with your Python version)
5. Place `GnuPG` and `lacre` directories in `/usr/local/lib/python3.x/dist-packages` (replace 3.x with your Python version). Make sure they're available for Python `import`s by executing `python -m lacre.admin -h` command.
6. Configure `/etc/gpg-mailgate.conf` based on the provided `gpg-mailgate.conf.sample`. Change the settings according to your configuration. If you follow this guide and have a standard configuration for postfix, you don't need to change much.
6. Configure `/etc/lacre.conf` based on the provided `lacre.conf.sample`. Change the settings according to your configuration. If you follow this guide and have a standard configuration for postfix, you don't need to change much.
7. Configure logging by copying `gpg-lacre-logging.conf.sample` to `/etc/gpg-lacre-logging.conf` and editing it according to your needs. The path to this file is included in `[logging]` section of `gpg-mailgate.conf` file, so if you place it somewhere else, make sure to update the path too. See also: [Configuration file format](https://docs.python.org/3/library/logging.config.html#configuration-file-format).
7. Configure logging by copying `lacre-logging.conf.sample` to `/etc/lacre-logging.conf` and editing it according to your needs. The path to this file is included in `[logging]` section of `lacre.conf` file, so if you place it somewhere else, make sure to update the path too. See also: Python logging package's [Configuration file format](https://docs.python.org/3/library/logging.config.html#configuration-file-format).
8. Add the following to the end of `/etc/postfix/master.cf`
```
gpg-mailgate unix - n n - - pipe
flags= user=nobody argv=/usr/local/bin/gpg-mailgate.py ${recipient}
lacre unix - n n - - pipe
flags= user=nobody argv=/usr/local/bin/lacre.py ${recipient}
127. 0. 0. 1:10028 inet n - n - 10 smtpd
-o content_filter=
@ -75,12 +73,12 @@ gpg-mailgate unix - n n - - pipe
-o smtpd_authorized_xforward_hosts=127. 0. 0. 0/8
```
If you use Postfix versions from 2.5 onwards, it is recommended to change `${recipient}` to `${original_recipient}` in line two of the lines above.
If you use Postfix versions from 2.5 onwards, it is recommended to change `${recipient}` to `${original_recipient}` in second line of the snippet above.
9. Add the following line to `/etc/postfix/main.cf`
```
content_filter = gpg-mailgate
content_filter = lacre
```
10. Optional: GPG can automatically download new public keys for automatic signature verification. To enable automatic create the file `/var/gpgmailgate/.gnupg/gpg.conf`. Add the following line to the file:
@ -97,23 +95,23 @@ You are now ready to go. To add a public key for encryption just use the followi
sudo -u nobody /usr/bin/gpg --homedir=/var/gpgmailgate/.gnupg --import /some/public.key
```
- Replace `/some/public.key` with the location of a public key
- `/some/public.key` can be deleted after importation
- Confirm that it's working:
`sudo -u nobody /usr/bin/gpg --list-keys --homedir=/var/gpgmailgate/.gnupg`
- Replace `/some/public.key` with the location of a public key (`/some/public.key` can be deleted after the import).
- Confirm that it's working: `sudo -u nobody /usr/bin/gpg --list-keys --homedir=/var/gpgmailgate/.gnupg`
If you already have a keyring you would like to import into Lacre, you can use `lacre.admin` command-line utility. Read more in [Lacre administration](doc/admin.md).
Please also test your installation before using it.
GPG-Mailgate is also able to handle S/MIME certificates for encrypting mails. However, it is best to use it in combination with Register-Handler described later to add new certificates. If you try to add them manually it might fail. The certificates are stored in `/var/gpgmailgate/smime` in PKCS7 format and are named like `User@example.com` (the user part is case sensitive, the domain part should be in lower case).
Lacre is also able to handle S/MIME certificates for encrypting mails. However, it is best to use it in combination with Register-Handler described later to add new certificates. If you try to add them manually it might fail. The certificates are stored in `/var/gpgmailgate/smime` in PKCS7 format and are named like `User@example.com` (the user part is case sensitive, the domain part should be in lower case).
#### Additional settings
Most mail servers do not handle mail addresses case sensitive. If you know that all your recipient mail servers do not care about case sensitivity then you can set `mail_case_insensitive` in the settings to `yes` so looking up PGP keys or S/MIME certificates does also happen case insensitive.
If your recipients have problems to decrypt mails encrypted by GPG-Mailgate they might use a piece of software that does not support PGP/MIME encrypted mails. You can tell GPG-Mailgate to use the legacy PGP/INLINE format by adding the recipient to the `pgp_style` map in the following format:
If your recipients have problems to decrypt mails encrypted by Lacre they might use a piece of software that does not support PGP/MIME encrypted mails. You can tell Lacre to use the legacy PGP/INLINE format by adding the recipient to the `pgp_style` map in the following format:
`User@example.com=inline`
## Install GPG-Mailgate-Web
## Install Lacre-Webgate
### Requirements
@ -124,7 +122,9 @@ If your recipients have problems to decrypt mails encrypted by GPG-Mailgate they
### Installation
All files you need can be found in the [gpg-mailgate-web](gpg-mailgate-web/) directory.
All files you need can be found in the
[Lacre / lacre-webgate](https://git.disroot.org/Lacre/lacre-webgate/)
repository.
1. Install the Python-mysqldb and Python-markdown modules:
@ -132,13 +132,13 @@ All files you need can be found in the [gpg-mailgate-web](gpg-mailgate-web/) dir
apt-get install python-mysqldb python-markdown
```
2. Create a new database for GPG-Mailgate-Web.
2. Create a new database for Lacre-Webgate.
3. Import the schema file `schema.sql` into the newly created database.
4. Edit the config file located at `/etc/gpg-mailgate.conf`. Set `enabled = yes` in `[database]` and fill in the necessary settings for the database connection.
4. Edit the config file located at `/etc/lacre.conf`. Set `enabled = yes` in `[database]` and fill in the necessary settings for the database connection.
5. Copy the files located in the [public_html](gpg-mailgate-web/public_html) directory onto your webserver. They can also be placed in a subdirectory on your webserver.
5. Copy the files located in the [public_html](https://git.disroot.org/Lacre/lacre-webgate/src/branch/main/public_html) directory onto your webserver. They can also be placed in a subdirectory on your webserver.
6. On your webserver move the `config.sample.php` file to `config.php` and edit the configuration file.
@ -154,28 +154,27 @@ mkdir -p /var/gpgmailgate/cron_templates
chown -R nobody:nogroup /var/gpgmailgate/cron_templates
```
9. Copy `cron.py` to `/usr/local/bin/gpgmw-cron.py`. Make it executable and and transfer ownership to `nobody`:
9. Copy `cron.py` to `/usr/local/bin/cron.py`. Make it executable and and transfer ownership to `nobody`:
```
chown nobody:nogroup /usr/local/bin/gpgmw-cron.py
chmod u+x /usr/local/bin/gpgmw-cron.py
install -u nobody -g nobody -m u+x cron.py /usr/local/bin/lacre-cron.py
```
10. Create `/etc/cron.d/gpgmw` with contents:
`*/3 * * * * nobody /usr/bin/python /usr/local/bin/gpgmw-cron.py > /dev/null`
10. Create `/etc/cron.d/lacre-cron` with contents:
`*/3 * * * * nobody /usr/bin/python /usr/local/bin/lacre-cron.py > /dev/null`
for executing the cron job automatically.
11. Test your installation.
### GPG-Mailgate-Web as keyserver
### Lacre-Webgate as keyserver
GPG-Mailgate-Web can also be used as a keyserver. For more information have a look at GPG-Mailgate-Web's [readme](gpg-mailgate-web/README).
Lacre-Webgate can also be used as a keyserver. For more information have a look at Lacre-Webgate's [README](https://git.disroot.org/Lacre/lacre-webgate/src/branch/main/README.md).
## Install Register-handler
### Requirements
- Already set up and working GPG-Mailgate-Web. It should be reachable from the machine that will run register-handler
- Already set up and working Lacre-Webgate. It should be reachable from the machine that will run register-handler
- Postfix is already installed and configured. It is recommended that you have already tested your configuration so we can exclude this as a main cause of problems. Your Postfix configuration should also support aliases
### Installation
@ -201,11 +200,10 @@ chown -R nobody:nogroup /var/gpgmailgate/register_templates
4. Copy `register-handler.py` to `/usr/local/bin/register-handler.py`. Make it executable and own it to `nobody`:
```
chown nobody:nogroup /usr/local/bin/register-handler.py
chmod a+x /usr/local/bin/register-handler.py
install -u nobody -g nogroup -m a+x register-handler.py /usr/local/bin/
```
5. Edit the config file located at `/etc/gpg-mailgate.conf`. Set the parameter `webpanel_url` in `[mailregister]` to the url of your GPG-Mailgate-Web panel (the URL should be the same as the one you use to access the panel with your web browser). Also set the parameter `register_email` to the email address you want the user to see when receiving mails from the register-handler (it does not have to be an existing address but it is recommended). Register-handler will send users mails when they are registering S/MIME certificates or when neither a S/MIME certificate nor a PGP key was found in a mail sent to the register-handler.
5. Edit the config file located at `/etc/lacre.conf`. Set the parameter `webpanel_url` in `[mailregister]` to the url of your Lacre-Webgate panel (the URL should be the same as the one you use to access the panel with your web browser). Also set the parameter `register_email` to the email address you want the user to see when receiving mails from the register-handler (it does not have to be an existing address but it is recommended). Register-handler will send users mails when they are registering S/MIME certificates or when neither a S/MIME certificate nor a PGP key was found in a mail sent to the register-handler.
6. Add `register: |/usr/local/bin/register-handler.py` to `/etc/aliases`

View File

@ -7,24 +7,30 @@
#
# make test PYTHON=/usr/local/bin/python3.8
#
# This marco is passed via environment to test/e2e_test.py, where it's
# This macro is passed via environment to test/e2e_test.py, where it's
# used to compute further commands.
#
PYTHON = python3
PYTHON = python
#
# SQLite database used during tests
#
# This database stores key queue and identity repository for e2etest,
# daemontest, and crontest.
#
TEST_DB = test/lacre.db
#
# Main goal to run tests.
# Main goal to run all tests.
#
test: e2etest unittest daemontest crontest
test: e2etest daemontest unittest crontest
#
# Run a set of end-to-end tests.
#
# Test scenarios are described and configured by the test/e2e.ini
# file. Basically this is just a script that feeds GPG Mailgate with
# known input and checks whether output meets expectations.
# Test scenarios are described and configured by the test/e2e.ini file.
# Basically this is just a script that feeds Lacre with known input and checks
# whether output meets expectations.
#
e2etest: test/tmp test/logs pre-clean restore-keyhome
$(PYTHON) test/e2e_test.py
@ -33,11 +39,12 @@ e2etest: test/tmp test/logs pre-clean restore-keyhome
# Run a basic cron-job test.
#
# We use PYTHONPATH to make sure that cron.py can import GnuPG
# package. We also set GPG_MAILGATE_CONFIG env. variable to make sure
# package. We also set LACRE_CONFIG env. variable to make sure
# it slurps the right config.
#
crontest: clean-db $(TEST_DB)
GPG_MAILGATE_CONFIG=test/gpg-mailgate-cron-test.conf PYTHONPATH=`pwd` $(PYTHON) webgate-cron.py
LACRE_CONFIG=test/lacre-daemon.conf PYTHONPATH=`pwd` \
$(PYTHON) webgate-cron.py
$(TEST_DB):
$(PYTHON) test/utils/schema.py $(TEST_DB)
@ -45,7 +52,7 @@ $(TEST_DB):
#
# Run an e2e test of Advanced Content Filter.
#
daemontest:
daemontest: restore-keyhome
$(PYTHON) test/daemon_test.py
# Before running the crontest goal we need to make sure that the
@ -57,14 +64,15 @@ clean-db:
# Run unit tests
#
unittest:
$(PYTHON) -m unittest discover -s test/modules
LACRE_CONFIG=test/lacre.conf $(PYTHON) -m unittest discover -s test/modules
pre-clean:
rm -fv test/gpg-mailgate.conf
rm -fv test/lacre.conf
rm -f test/logs/*.log
restore-keyhome:
git restore test/keyhome
git restore test/keyhome.other
test/tmp:
mkdir test/tmp
@ -72,5 +80,5 @@ test/tmp:
test/logs:
mkdir test/logs
clean: pre-clean
clean: pre-clean clean-db
rm -rfv test/tmp test/logs

View File

@ -1,57 +1,68 @@
# GPG Lacre Project
# Lacre Project
GPG Lacre is a fork and continuation of original work of gpg-mailgate project:
[gpg-mailgate](https://github.com/TheGreatGooo/gpg-mailgate). It is still
actively developed and should be considered as beta -- with all APIs and
internals being subject to change. Please only use this software if you know
GnuPG well.
**Lacre** (wax seal in Portuguese) is an add-on for Postfix that automatically
encrypts incoming email before delivering it to recipients' inbox for
recipients that have provided their public keys.
Lacre is a fork and continuation of the original work on
[gpg-mailgate](https://github.com/TheGreatGooo/gpg-mailgate) project. It is
still actively developed and should be considered as beta -- with all APIs and
internals being subject to change. Please only use this software if you know
GnuPG well and accept occasional failures.
**GPG Lacre** (wax seal in Portuguese) is a content filter for Postfix that automatically encrypts unencrypted incoming email using PGP or S/MIME for select recipients.
This project is the continuation of the work of "gpg-mailgate" on providing open source, GnuPG based email encryption for emails at rest. All incoming emails are automatically encrypted with user's public key before they are saved on the server. It is a server side encryption solution while the control of the encryption keys are fully at the hands of the end-user and private keys are never stored on the server.
# How it works
The scope of the project is to improve on the already existing code, provide easy to use key upload system (standalone as well as Roundcube plugin) and key discoverability. Beside providing a solution that is easy to use we will also provide easy to digest material about encryption, how it works and how to make use of it in situations other the just mailbox encryption. Understanding how encryption works is the key to self-determination and is therefore an important part of the project.
Lacre is a [content filter](https://www.postfix.org/FILTER_README.html). This
means, that when Postfix receives a message, it "forwards" that message to
Lacre and if Lacre delivers it to a given destination, the message arrives to
recipient's inbox.
GPG Lacre will be battle tested on the email infrastructure of https://disroot.org (an ethical non-profit service provider).
After receiving the message, Lacre does the following:
1. If message already is encrypted, it just delivers the message immediately.
2. Checks the list of recipients, finds their public keys if any were
provided.
3. Encrypts message if possible.
4. Delivers the message.
---
The work on this project in 2021 is funded by https://nlnet.nl/thema/NGIZeroPET.html for which we are very thankful.
The scope of the work for 2021 is:
- Rewrite code to python3
- Improve standalone key upload website
- Provide Roundcube plugin for key management
- Improve key server features
- Provide webiste with information and tutorials on how to use GPG in general and also **Lacre**
- (Optional) provide Autocrypt support
Work on this project in 2021 was funded by
[NGI Zero PET](https://nlnet.nl/thema/NGIZeroPET.html)
for which we are very thankful.
Made possible thanks to:<br>
![](https://nlnet.nl/logo/banner.png)
---
For installation instructions, please refer to the included **INSTALL** file.
# Installation
For installation instructions, please refer to the included [INSTALL](INSTALL.md) file.
---
# Features
# Planned features
- Correctly displays attachments and general email content; currently will only display first part of multipart messages
- Public keys are stored in a dedicated gpg-home-directory
- Encrypts both matching incoming and outgoing mail (this means gpg-mailgate can be used to encrypt outgoing mail for software that doesn't support PGP or S/MIME)
- Decrypts PGP encrypted mails for present private keys (but no signature check and it does not always work with PGP/INLINE encrypted mails)
- Easy installation
- gpg-mailgate-web extension is a web interface allowing any user to upload PGP keys so that emails sent to them from your mail server will be encrypted (see gpg-mailgate-web directory for details)
- people can submit their public key like to any keyserver to gpg-mailgate with the gpg-mailgate-web extension
- people can send an S/MIME signed email to register@yourdomain.tld to register their public key
- people can send their public OpenPGP key as attachment or inline to register@yourdomain.tld to register it
- People can submit their public key like to any keyserver to gpg-mailgate with the gpg-mailgate-web extension
- People can send an S/MIME signed email to register@yourdomain.tld to register their public key
- People can send their public OpenPGP key as attachment or inline to register@yourdomain.tld to register it
See also: [lacre-webgate](https://git.disroot.org/Lacre/lacre-webgate/) -- a
web interface allowing any user to upload PGP keys so that emails sent to them
from your mail server will be encrypted
This is forked from the original project at http://code.google.com/p/gpg-mailgate/
# Authors
This is a combined work of many developers and contributors. We would like to pay honours to original gpg mailbox developers for making this project happen, and providing solid solution for encryption emails at rest:
This is a combined work of many developers and contributors. We would like to
pay honours to original gpg mailbox developers for making this project happen,
and providing solid solution for encryption emails at rest:
* mcmaster <mcmaster@aphrodite.hurricanelabs.rsoc>
* Igor Rzegocki <ajgon@irgon.com> - [GitHub](https://github.com/ajgon/gpg-mailgate)
@ -63,4 +74,4 @@ This is a combined work of many developers and contributors. We would like to pa
* Bruce Markey - [GitHub](https://github.com/TheEd1tor)
* Remko Tronçon - [GitHub](https://github.com/remko/phkp/)
* Kiritan Flux [GitHub](https://github.com/kflux)
* Fabian Krone [GitHub] (https://github.com/fkrone/gpg-mailgate)
* Fabian Krone [GitHub](https://github.com/fkrone/gpg-mailgate)

21
bin/lacreadm Executable file
View File

@ -0,0 +1,21 @@
#!/bin/sh
#
# lacre
#
# This file is part of the lacre source code.
#
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
python -m lacre.admin $*

77
doc/admin.md Normal file
View File

@ -0,0 +1,77 @@
# Lacre administration
## Command-line tool
There's a little tool for administrators. As long as Lacre Python packages
are available via `PYTHONPATH`, you can use it like this:
```sh
python -m lacre.admin -h
```
Of course `-h` displays some help.
**Note:** Help output includes information about the configuration file being
in use, which may be useful at times.
**Note:** You can also use a tiny shell wrapper around this tool, see
`bin/lacreadm`.
## Initialising database schema
If you want to initialise Lacre's database (which is also used by the
frontend), run:
```sh
python -m lacre.admin database -i
```
## Inspecting key confirmation queue
To find out how many keys are waiting to be confirmed, run:
```sh
python -m lacre.admin queue
```
To see identities (emails) waiting for confirmation, use `--list` (or `-l`)
option:
```sh
python -m lacre.admin queue -l
```
To delete one of these emails, use `--delete` (or `-d`) option:
```sh
python -m lacre.admin queue -d malory@example.org
```
## Inspecting identities registered
To list all identities, run:
```sh
python -m lacre.admin identities -a
```
To preview a particular identity, run:
```sh
python -m lacre.admin identities -e alice@example.com
```
## Importing identities from existing GnuPG keyring
If you already have a GnuPG keyring with your users' public keys or for some
reason Lacre's identity database needs to be re-populated with identities,
there's a command to do that:
```sh
python -m lacre.admin import -d /path/to/gnupg/directory
```
If you want to just re-populate the database, Lacre can remove all identities
prior to importing keys -- just add `-r` flag.

View File

@ -33,13 +33,13 @@ setting port to `10026`.
Command to spawn a Lacre daemon process is:
```
GPG_MAILGATE_CONFIG=/etc/gpg-mailgate.conf PYTHONPATH=... python -m lacre.daemon
LACRE_CONFIG=/etc/lacre.conf PYTHONPATH=... python -m lacre.daemon
```
Two environment variables used here are:
* `GPG_MAILGATE_CONFIG` (not mandatory) -- path to Lacre configuration,
unless it's kept in default location (`/etc/gpg-maillgate.conf`).
* `LACRE_CONFIG` (not mandatory) -- path to Lacre configuration,
unless it's kept in default location (`/etc/lacre.conf`).
* `PYTHONPATH` (not mandatory) -- location of Lacre modules. You can place
them below your Python's `site-packages` to be reachable by any other
Python software.

View File

@ -49,3 +49,11 @@ verifying that the correct key has been used. That's because we don't know
When things go wrong, be sure to study `test/logs/e2e.log` and
`test/logs/gpg-mailgate.log` files -- they contain some useful information.
## Test identities
There are several identities in test/keyhome and in the test database:
* alice@disposlab: 1CD245308F0963D038E88357973CF4D9387C44D7
* bob@disposlab: 19CF4B47ECC9C47AFA84D4BD96F39FDA0E31BB67
* evan@disposlab: 530B1BB2D0CC7971648198BBA4774E507D3AF5BC

View File

@ -1,69 +0,0 @@
#!/usr/bin/python
#
# gpg-mailgate
#
# This file is part of the gpg-mailgate source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
#
import email
from email.policy import SMTPUTF8
import sys
import time
import logging
import lacre
import lacre.config as conf
start = time.process_time()
conf.load_config()
lacre.init_logging(conf.get_item('logging', 'config'))
# This has to be executed *after* logging initialisation.
import lacre.core as core
LOG = logging.getLogger('gpg-mailgate.py')
missing_params = conf.validate_config()
if missing_params:
LOG.error(f"Aborting delivery! Following mandatory config parameters are missing: {missing_params!r}")
sys.exit(lacre.EX_CONFIG)
delivered = False
try:
# Read e-mail from stdin, parse it
raw = sys.stdin.read()
raw_message = email.message_from_string(raw, policy=SMTPUTF8)
from_addr = raw_message['From']
# Read recipients from the command-line
to_addrs = sys.argv[1:]
# Let's start
core.deliver_message(raw_message, from_addr, to_addrs)
process_t = (time.process_time() - start) * 1000
LOG.info("Message delivered in {process:.2f} ms".format(process=process_t))
delivered = True
except:
LOG.exception('Could not handle message')
if not delivered:
# It seems we weren't able to deliver the message. In case it was some
# silly message-encoding issue that shouldn't bounce the message, we just
# try recoding the message body and delivering it.
try:
core.failover_delivery(raw_message, to_addrs, from_addr)
except:
LOG.exception('Failover delivery failed too')

View File

@ -1,11 +1,11 @@
[default]
# Whether gpg-mailgate should add a header after it has processed an email
# Whether lacre should add a header after it has processed an email
# This may be useful for debugging purposes
add_header = yes
# Whether we should only encrypt emails if they are explicitly defined in
# the key mappings below ([enc_keymap] section)
# This means gpg-mailgate won't automatically detect PGP recipients for encrypting
# This means lacre won't automatically detect PGP recipients for encrypting
enc_keymap_only = no
# Convert encrypted text/plain email to MIME-attached encrypt style.
@ -20,36 +20,36 @@ mime_conversion = yes
mail_case_insensitive = no
[gpg]
# the directory where gpg-mailgate public keys are stored
# the directory where lacre public keys are stored
# (see INSTALL for details)
#
# Note that this directory should be accessible only for the Lacre user,
# i.e. have mode 700.
keyhome = /var/gpgmailgate/.gnupg
keyhome = /var/lacre/.gnupg
[smime]
# the directory for the S/MIME certificate files
cert_path = /var/gpgmailgate/smime
cert_path = /var/lacre/smime
[mailregister]
# settings for the register-handler
register_email = register@yourdomain.tld
mail_templates = /var/gpgmailgate/register_templates
mail_templates = /var/lacre/register_templates
# URL to webpanel. Upon receiving an email with a key, register-handler
# uploads it to the web panel.
webpanel_url = http://yourdomain.tld
[cron]
# settings for the gpgmw cron job
# settings for the cron job
send_email = yes
notification_email = gpg-mailgate@yourdomain.tld
mail_templates = /var/gpgmailgate/cron_templates
notification_email = lacre@yourdomain.tld
mail_templates = /var/lacre/cron_templates
[logging]
# path to the logging configuration; see documentation for details:
# https://docs.python.org/3/library/logging.config.html#logging-config-fileformat
config = /etc/gpg-lacre-logging.conf
config = /etc/lacre-logging.conf
[daemon]
# Advanced Content Filter section.
@ -68,14 +68,19 @@ max_data_bytes = 33554432
# This should never be PII, but information like encoding, content types, etc.
log_headers = no
# Sometimes we might fail to load keys and need to choose between delivering
# in cleartext or not delivering. The default is to deliver cleartext, but
# administrators can make this decision on their own.
bounce_on_keys_missing = no
[relay]
# the relay settings to use for Postfix
# gpg-mailgate will submit email to this relay after it is done processing
# lacre will submit email to this relay after it is done processing
# unless you alter the default Postfix configuration, you won't have to modify this
host = 127.0.0.1
port = 10028
# This is the default port of postfix. It is used to send some
# mails through the GPG-Mailgate so they are encrypted
# mails through the Lacre so they are encrypted
enc_port = 25
# Set this option to yes to use TLS for SMTP Servers which require TLS.
@ -84,7 +89,7 @@ starttls = no
[smtp]
# Options when smtp auth is required to send out emails
enabled = false
username = gpg-mailgate
username = lacre
password = changeme
host = yourdomain.tld
port = 587
@ -92,18 +97,42 @@ starttls = true
[database]
# edit the settings below if you want to read keys from a
# gpg-mailgate-web database other than SQLite
# lacre-webgate database other than SQLite
enabled = yes
url = sqlite:///test.db
# For a MySQL database "gpgmw", user "gpgmw" and password "password",
# Pooling mode: pessimistic or optimistic (required parameter).
#
# - Pessimistic disconnect-handling: pre_ping. Connection pool will try using
# connection before it executes a SQL query to find out if the connection is
# still alive. If not, it'll just establish a new connection.
#
# - Optimistic distonnect-handling: just avoid using connections after some
# time.
#
pooling_mode = optimistic
# For a MySQL database "lacre", user "lacre" and password "password",
# use the following URL:
#
#url = mysql://gpgmw:password@localhost/gpgmw
#url = mysql://lacre:password@localhost/lacre
#
# For other RDBMS backends, see:
# https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls
# Number of seconds after which an idle connection is recycled. This is
# useful with MySQL servers. This is only used with pooling_mode=optimistic.
# For more information, see:
# https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine.params.pool_recycle
#max_connection_age = 3600
# Number of connections stored in the pool.
#pool_size = 5
# If the pool size is not enough for current traffic, some connections can be
# made and closed after use, to avoid pool growth and connection rejections.
#max_overflow = 10
[enc_keymap]
# You can find these by running the following command:
# gpg --list-keys --keyid-format long user@example.com

73
lacre.py Executable file
View File

@ -0,0 +1,73 @@
#!/usr/bin/python
#
# lacre
#
# This file is part of the lacre source code.
#
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
import email
from email.policy import SMTPUTF8
import sys
import logging
import lacre
import lacre.config as conf
from lacre.stats import time_logger
conf.load_config()
lacre.init_logging(conf.get_item('logging', 'config'))
# This has to be executed *after* logging initialisation.
import lacre.core as core
LOG = logging.getLogger('lacre.py')
def main():
with time_logger('Message delivery', LOG):
missing_params = conf.validate_config()
config_file = conf.config_source()
if missing_params:
LOG.error(f"Aborting delivery! Following mandatory config parameters are missing in {config_file!r}: {missing_params}")
sys.exit(lacre.EX_CONFIG)
delivered = False
try:
# Read e-mail from stdin, parse it
raw = sys.stdin.read()
raw_message = email.message_from_string(raw, policy=SMTPUTF8)
from_addr = raw_message['From']
# Read recipients from the command-line
to_addrs = sys.argv[1:]
# Let's start
core.deliver_message(raw_message, from_addr, to_addrs)
delivered = True
except:
LOG.exception('Could not handle message')
if not delivered:
# It seems we weren't able to deliver the message. In case it was
# some silly message-encoding issue that shouldn't bounce the
# message, we just try recoding the message body and delivering it.
try:
core.failover_delivery(raw_message, to_addrs, from_addr)
except:
LOG.exception('Failover delivery failed too')
if __name__ == '__main__':
main()

View File

@ -8,30 +8,30 @@ import logging.config
# be performed. It only sets up a syslog handler, so that the admin has at
# least some basic information.
FAIL_OVER_LOGGING_CONFIG = {
'version': 1,
'formatters': {
'sysfmt': {
'format': '%(asctime)s %(module)s %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
},
'handlers': {
'syslog': {
'class': 'logging.handlers.SysLogHandler',
'level': 'INFO',
'formatter': 'sysfmt'
},
'lacrelog': {
'class': 'logging.FileHandler',
'level': 'INFO',
'formatter': 'sysfmt',
'filename': 'lacre.log'
}
},
'root': {
'level': 'INFO',
'handlers': ['syslog', 'lacrelog']
}
'version': 1,
'formatters': {
'sysfmt': {
'format': '%(asctime)s %(module)s %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S'
},
},
'handlers': {
'syslog': {
'class': 'logging.handlers.SysLogHandler',
'level': 'INFO',
'formatter': 'sysfmt'
},
'lacrelog': {
'class': 'logging.FileHandler',
'level': 'INFO',
'formatter': 'sysfmt',
'filename': 'lacre.log'
}
},
'root': {
'level': 'INFO',
'handlers': ['syslog', 'lacrelog']
}
}
# Exit code taken from <sysexits.h>:
@ -41,8 +41,9 @@ EX_CONFIG = 78
def init_logging(config_filename):
if config_filename is not None:
logging.config.fileConfig(config_filename)
else:
logging.config.dictConfig(FAIL_OVER_LOGGING_CONFIG)
logging.warning('Lacre logging configuration missing, using syslog as default')
if config_filename is not None:
logging.config.fileConfig(config_filename)
logging.info('Configured from %s', config_filename)
else:
logging.config.dictConfig(FAIL_OVER_LOGGING_CONFIG)
logging.warning('Lacre logging configuration missing, using syslog as default')

63
lacre/_keyringcommon.py Normal file
View File

@ -0,0 +1,63 @@
class KeyCache:
"""A store for OpenPGP keys.
Key case is sanitised while loading from GnuPG if so
configured. See mail_case_insensitive parameter in section
[default].
"""
def __init__(self, keys: dict = None):
"""Initialise an empty cache.
With keyring_dir given, set location of the directory from which keys should be loaded.
"""
self._keys = keys
def __getitem__(self, fingerpring):
"""Look up email assigned to the given fingerprint."""
return self._keys[fingerpring]
def __setitem__(self, fingerprint, email):
"""Assign an email to a fingerpring, overwriting it if it was already present."""
self._keys[fingerprint] = email
def __contains__(self, fingerprint):
"""Check if the given fingerprint is assigned to an email."""
# This method has to be present for KeyCache to be a dict substitute.
# See mailgate, function _identify_gpg_recipients.
return fingerprint in self._keys
def has_email(self, email):
"""Check if cache contains a key assigned to the given email."""
return email in self._keys.values()
def __repr__(self):
"""Return text representation of this object."""
details = ' '.join(self._keys.keys())
return '<KeyCache %s>' % (details)
def __iter__(self):
return iter(self._keys.keys())
def emails(self):
return { email: fingerprint for (fingerprint, email) in self._keys.items() }
class KeyRing:
"""Contract to be implemented by a key-store (a.k.a. keyring)."""
def freeze_identities(self) -> KeyCache:
"""Return a static, async-safe copy of the identity map."""
raise NotImplementedError('KeyRing.load not implemented')
def register_or_update(self, email: str, key_id: str):
"""Add a new (email,key) pair to the keystore."""
raise NotImplementedError('KeyRing.register_or_update not implemented')
def post_init_hook(self):
"""Lets the keyring perform additional operations following its initialisation."""
pass
def shutdown(self):
"""Lets the keyring perform operations prior to shutting down."""
pass

168
lacre/admin.py Normal file
View File

@ -0,0 +1,168 @@
"""Lacre administrative tool.
This is a command-line tool expected to be run by a person who knows what they
are doing. Also, please read the docs first.
"""
import sys
import argparse
import logging
import GnuPG
import lacre
import lacre.config as conf
conf.load_config()
lacre.init_logging(conf.get_item('logging', 'config'))
import lacre.repositories as repo
import lacre.dbschema as db
if __name__ == '__main__':
LOG = logging.getLogger('lacre.admin')
else:
LOG = logging.getLogger(__name__)
def _no_database():
print('Database unavailable or not configured properly')
sys.exit(lacre.EX_CONFIG)
def sub_db(args):
"""Sub-command to manipulate database."""
LOG.debug('Database operations ahead')
if args.init:
eng = repo.init_engine(conf.get_item('database', 'url'))
LOG.warning('Initialising database schema with engine: %s', eng)
print('Creating database tables')
db.create_tables(eng)
def sub_queue(args):
"""Sub-command to inspect queue contents."""
LOG.debug('Inspecting queue...')
eng = repo.init_engine(conf.get_item('database', 'url'))
queue = repo.KeyConfirmationQueue(engine=eng)
if args.delete:
queue.delete_key_by_email(args.delete)
elif args.list:
for k in queue.fetch_keys():
print(f'- {k.id}: {k.email}')
elif args.to_delete:
for k in queue.fetch_keys_to_delete():
print(f'- {k.id}: {k.email}')
else:
cnt = queue.count_keys()
if cnt is None:
_no_database()
print(f'Keys in the queue: {cnt}')
def sub_identities(args):
"""Sub-command to inspect identity database."""
LOG.debug('Inspecting identities...')
eng = repo.init_engine(conf.get_item('database', 'url'))
identities = repo.IdentityRepository(engine=eng)
all_identities = identities.freeze_identities()
if all_identities is None:
_no_database()
if args.email:
all_rev = all_identities.emails()
print('-', args.email, all_rev[args.email])
else:
for id_ in all_identities:
print('-', all_identities[id_], id_)
def sub_import(args):
"""Sub-command to import all identities known to GnuPG into Lacre database."""
LOG.debug('Importing identities...')
source_dir = args.homedir or conf.get_item('gpg', 'keyhome')
public = GnuPG.public_keys(source_dir)
eng = repo.init_engine(conf.get_item('database', 'url'))
identities = repo.IdentityRepository(engine=eng)
if args.reload:
identities.delete_all()
total = 0
for (fingerprint, email) in public.items():
LOG.debug('Importing %s - %s', email, fingerprint)
identities.register_or_update(email, fingerprint)
total += 1
LOG.debug('Imported %d identities', total)
print(f'Imported {total} identities')
def main():
missing = conf.validate_config()
if missing:
LOG.error('Missing configuration parameters: %s', missing)
print('Insufficient configuration, aborting.')
sys.exit(lacre.EX_CONFIG)
general_conf = conf.config_source()
log_conf = conf.get_item('logging', 'config')
parser = argparse.ArgumentParser(
prog='lacre.admin',
description='Lacre Admin\'s best friend',
epilog=f'Config read from {general_conf}. For diagnostic info, see {log_conf}'
)
sub_commands = parser.add_subparsers(help='Sub-commands', required=True)
cmd_db = sub_commands.add_parser('database',
help='',
aliases=['db']
)
cmd_db.add_argument('-i', '--init', action='store_true',
help='Initialise database schema')
cmd_db.set_defaults(operation=sub_db)
cmd_import = sub_commands.add_parser('import',
help='Load identities from GnuPG directory to Lacre database'
)
cmd_import.add_argument('-d', '--homedir', default=False,
help='specify GnuPG directory (default: use configured dir.)')
cmd_import.add_argument('-r', '--reload', action='store_true',
help='delete all keys from database before importing')
cmd_import.set_defaults(operation=sub_import)
cmd_queue = sub_commands.add_parser('queue',
help='Inspect key queue',
aliases=['q']
)
cmd_queue.add_argument('-D', '--delete',
help='delete specified email from the queue')
cmd_queue.add_argument('-l', '--list', action='store_true',
help='list keys in the queue')
cmd_queue.add_argument('-d', '--to-delete', action='store_true',
help='list keys to be deleted')
cmd_queue.set_defaults(operation=sub_queue)
cmd_identities = sub_commands.add_parser('identities',
help='Inspect identity database',
aliases=['id']
)
cmd_identities.add_argument('-e', '--email', help='look up a single email')
cmd_identities.set_defaults(operation=sub_identities)
user_request = parser.parse_args()
user_request.operation(user_request)
if __name__ == '__main__':
main()

View File

@ -4,15 +4,16 @@ Routines defined here are responsible for processing and validating
configuration.
"""
from enum import Enum, auto
from configparser import RawConfigParser
import os
# Environment variable name we read to retrieve configuration path. This is to
# enable non-root users to set up and run GPG Mailgate and to make the software
# enable non-root users to set up and run Lacre and to make the software
# testable.
CONFIG_PATH_ENV = "GPG_MAILGATE_CONFIG"
CONFIG_PATH_ENV = "LACRE_CONFIG"
# List of mandatory configuration parameters. Each item on this list should be
# a pair: a section name and a parameter name.
@ -20,7 +21,12 @@ MANDATORY_CONFIG_ITEMS = [("relay", "host"),
("relay", "port"),
("daemon", "host"),
("daemon", "port"),
("gpg", "keyhome")]
("gpg", "keyhome"),
('database', 'enabled'),
('database', 'url'),
('database', 'pooling_mode')]
CRON_REQUIRED = [('cron', 'mail_templates')]
# Global dict to keep configuration parameters. It's hidden behind several
# utility functions to make it easy to replace it with ConfigParser object in
@ -34,11 +40,11 @@ def load_config() -> dict:
If environment variable identified by CONFIG_PATH_ENV
variable is set, its value is taken as a configuration file
path. Otherwise, the default is taken
('/etc/gpg-mailgate.conf').
('/etc/lacre.conf').
"""
configFile = os.getenv(CONFIG_PATH_ENV, '/etc/gpg-mailgate.conf')
config_file = config_source()
parser = _read_config(configFile)
parser = _read_config(config_file)
# XXX: Global variable. It is a left-over from old GPG-Mailgate code. We
# should drop it and probably use ConfigParser instance where configuration
@ -48,6 +54,14 @@ def load_config() -> dict:
return cfg
def config_source() -> str:
"""Return path of configuration file.
Taken from LACRE_CONFIG environment variable, and if it's not
set, defaults to /etc/lacre.conf."""
return os.getenv(CONFIG_PATH_ENV, '/etc/lacre.conf')
def _read_config(fileName) -> RawConfigParser:
cp = RawConfigParser()
cp.read(fileName)
@ -90,16 +104,23 @@ def flag_enabled(section, key) -> bool:
return config_item_equals(section, key, 'yes')
def validate_config():
def validate_config(*, additional=None):
"""Check if configuration is complete.
Returns a list of missing parameters, so an empty list means
configuration is complete.
If 'additional' parameter is specified, it should be a list of
tuples (section, param).
"""
missing = []
for (section, param) in MANDATORY_CONFIG_ITEMS:
if not config_item_set(section, param):
missing.append((section, param))
if additional:
for (section, param) in additional:
if not config_item_set(section, param):
missing.append((section, param))
return missing
@ -120,3 +141,48 @@ def daemon_params():
def strict_mode():
"""Check if Lacre is configured to support only a fixed list of keys."""
return ("default" in cfg and cfg["default"]["enc_keymap_only"] == "yes")
def should_log_headers() -> bool:
"""Check if Lacre should log message headers."""
return flag_enabled('daemon', 'log_headers')
class FromStrMixin:
"""Additional operations for configuration enums."""
@classmethod
def from_str(cls, name, *, required=False):
if name is None:
return None
name = name.upper()
if name in cls.__members__:
return cls.__members__[name]
if required:
raise NameError('Unsupported or missing value')
else:
return None
@classmethod
def from_config(cls, section, key, *, required=False):
param = get_item(section, key)
return cls.from_str(param, required=required)
class PGPStyle(FromStrMixin, Enum):
"""PGP message structure: PGP/Inline or PGP/MIME."""
MIME = auto()
INLINE = auto()
class PoolingMode(FromStrMixin, Enum):
"""Database connection pool behaviour.
- Optimistic - recycles connections.
- Pessimistic - checks connection before usage.
"""
OPTIMISTIC = auto()
PESSIMISTIC = auto()

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
"""Lacre's actual mail-delivery module.
@ -40,7 +40,7 @@ import lacre.keyring as kcache
import lacre.recipients as recpt
import lacre.smime as smime
from lacre.transport import send_msg, register_sender, SendFrom
from lacre.mailop import KeepIntact, InlineOpenPGPEncrypt, MimeOpenPGPEncrypt
from lacre.mailop import KeepIntact, InlineOpenPGPEncrypt, MimeOpenPGPEncrypt, MailSerialisationException
LOG = logging.getLogger(__name__)
@ -88,17 +88,18 @@ def _sort_gpg_recipients(gpg_to) -> Tuple[recpt.RecipientList, recpt.RecipientLi
for rcpt in gpg_to:
# Checking pre defined styles in settings first
if conf.config_item_equals('pgp_style', rcpt.email(), 'mime'):
style = conf.PGPStyle.from_config('pgp_style', rcpt.email())
if style is conf.PGPStyle.MIME:
recipients_mime.append(rcpt.email())
keys_mime.extend(rcpt.key().split(','))
elif conf.config_item_equals('pgp_style', rcpt.email(), 'inline'):
elif style is conf.PGPStyle.INLINE:
recipients_inline.append(rcpt.email())
keys_inline.extend(rcpt.key().split(','))
else:
# Log message only if an unknown style is defined
if conf.config_item_set('pgp_style', rcpt.email()):
LOG.debug("Style %s for recipient %s is not known. Use default as fallback."
% (conf.get_item("pgp_style", rcpt.email()), rcpt.email()))
LOG.debug("Style %s for recipient %s is not known. Use default as fallback.",
conf.get_item("pgp_style", rcpt.email()), rcpt.email())
# If no style is in settings defined for recipient, use default from settings
if default_to_pgp_mime:
@ -126,7 +127,10 @@ def _gpg_encrypt_copy(message: EmailMessage, keys, recipients, encrypt_f):
def _gpg_encrypt_to_bytes(message: EmailMessage, keys, recipients, encrypt_f) -> bytes:
msg_copy = _gpg_encrypt_copy(message, keys, recipients, encrypt_f)
return msg_copy.as_bytes(policy=SMTPUTF8)
try:
return msg_copy.as_bytes(policy=SMTPUTF8)
except IndexError:
raise MailSerialisationException()
def _gpg_encrypt_to_str(message: EmailMessage, keys, recipients, encrypt_f) -> str:
@ -141,7 +145,7 @@ def _gpg_encrypt_and_deliver(message: EmailMessage, keys, recipients, encrypt_f)
def _customise_headers(message: EmailMessage):
if conf.flag_enabled('default', 'add_header'):
message['X-GPG-Mailgate'] = 'Encrypted by GPG Mailgate'
message['X-Lacre'] = 'Encrypted by Lacre'
def _encrypt_all_payloads_inline(message: EmailMessage, gpg_to_cmdline):
@ -209,7 +213,7 @@ def _rewrap_payload(message: EmailMessage) -> MIMEPart:
wrapper.set_type(message.get_content_type())
# Copy all Content-Type parameters.
for (pname, pvalue) in message.get_params():
for (pname, pvalue) in message.get_params(failobj=list()):
# Skip MIME type that's also returned by get_params().
if not '/' in pname:
wrapper.set_param(pname, pvalue)

View File

@ -3,6 +3,7 @@
import logging
import lacre
from lacre.text import DOUBLE_EOL_BYTES
from lacre.stats import time_logger
import lacre.config as conf
import sys
from aiosmtpd.controller import Controller
@ -10,11 +11,9 @@ from aiosmtpd.smtp import Envelope
import asyncio
import email
from email.policy import SMTPUTF8
import time
from watchdog.observers import Observer
# Load configuration and init logging, in this order. Only then can we load
# the last Lacre module, i.e. lacre.mailgate.
# the last Lacre module, i.e. lacre.core.
conf.load_config()
lacre.init_logging(conf.get_item("logging", "config"))
LOG = logging.getLogger('lacre.daemon')
@ -23,7 +22,7 @@ from GnuPG import EncryptionException
import lacre.core as gate
import lacre.keyring as kcache
import lacre.transport as xport
from lacre.mailop import KeepIntact
from lacre.mailop import KeepIntact, MailSerialisationException
class MailEncryptionProxy:
@ -35,50 +34,42 @@ class MailEncryptionProxy:
async def handle_DATA(self, server, session, envelope: Envelope):
"""Accept a message and either encrypt it or forward as-is."""
start = time.process_time()
try:
keys = await self._keyring.freeze_identities()
LOG.debug('Parsing message: %s', self._beginning(envelope))
message = email.message_from_bytes(envelope.original_content, policy=SMTPUTF8)
LOG.debug('Parsed into %s: %s', type(message), repr(message))
with time_logger('Message delivery', LOG):
try:
keys = self._keyring.freeze_identities()
message = email.message_from_bytes(envelope.original_content, policy=SMTPUTF8)
if message.defects:
# Sometimes a weird message cannot be encoded back and
# delivered, so before bouncing such messages we at least
# record information about the issues. Defects are identified
# by email.* package.
LOG.warning("Issues found: %d; %s", len(message.defects), repr(message.defects))
if message.defects:
LOG.warning("Issues found: %d; %s", len(message.defects), repr(message.defects))
if conf.flag_enabled('daemon', 'log_headers'):
LOG.info('Message headers: %s', self._extract_headers(message))
send = xport.SendFrom(envelope.mail_from)
for operation in gate.delivery_plan(envelope.rcpt_tos, message, keys):
LOG.debug(f"Sending mail via {operation!r}")
try:
new_message = operation.perform(message)
send(new_message, operation.recipients())
except (EncryptionException, MailSerialisationException, UnicodeEncodeError):
# If the message can't be encrypted, deliver cleartext.
LOG.exception('Unable to encrypt message, delivering in cleartext')
if not isinstance(operation, KeepIntact):
self._send_unencrypted(operation, envelope, send)
else:
LOG.exception('Cannot perform: %s', operation)
raise
send = xport.SendFrom(envelope.mail_from)
for operation in gate.delivery_plan(envelope.rcpt_tos, message, keys):
LOG.debug(f"Sending mail via {operation!r}")
try:
new_message = operation.perform(message)
send(new_message, operation.recipients())
except EncryptionException:
# If the message can't be encrypted, deliver cleartext.
LOG.exception('Unable to encrypt message, delivering in cleartext')
if not isinstance(operation, KeepIntact):
self._send_unencrypted(operation, message, envelope, send)
else:
LOG.error(f'Cannot perform {operation}')
except:
LOG.exception('Unexpected exception caught, bouncing message')
except:
LOG.exception('Unexpected exception caught, bouncing message')
return xport.RESULT_ERROR
if conf.should_log_headers():
LOG.error('Erroneous message headers: %s', self._beginning(envelope))
ellapsed = (time.process_time() - start) * 1000
LOG.info(f'Message delivered in {ellapsed:.2f} ms')
return xport.RESULT_ERRORR
return xport.RESULT_OK
def _send_unencrypted(self, operation, message, envelope, send: xport.SendFrom):
keep = KeepIntact(operation.recipients())
new_message = keep.perform(message)
send(new_message, operation.recipients(), envelope.mail_from)
def _send_unencrypted(self, operation, envelope, send: xport.SendFrom):
# Do not parse and re-generate the message, just send it as it is.
send(envelope.original_content, operation.recipients())
def _beginning(self, e: Envelope) -> bytes:
double_eol_pos = e.original_content.find(DOUBLE_EOL_BYTES)
@ -89,12 +80,8 @@ class MailEncryptionProxy:
end = min(limit, 2560)
return e.original_content[0:end]
def _extract_headers(self, message: email.message.Message):
return {
'mime' : message.get_content_type(),
'charsets' : message.get_charsets(),
'cte' : message['Content-Transfer-Encoding']
}
def _seconds_between(self, start_ms, end_ms) -> float:
return (end_ms - start_ms) * 1000
def _init_controller(keys: kcache.KeyRing, max_body_bytes=None, tout: float = 5):
@ -106,13 +93,6 @@ def _init_controller(keys: kcache.KeyRing, max_body_bytes=None, tout: float = 5)
data_size_limit=max_body_bytes)
def _init_reloader(keyring_dir: str, reloader) -> kcache.KeyringModificationListener:
listener = kcache.KeyringModificationListener(reloader)
observer = Observer()
observer.schedule(listener, keyring_dir, recursive=False)
return observer
def _validate_config():
missing = conf.validate_config()
if missing:
@ -130,7 +110,7 @@ async def _sleep():
await asyncio.sleep(360)
def _main():
async def _main():
_validate_config()
keyring_path = conf.get_item('gpg', 'keyhome')
@ -138,30 +118,30 @@ def _main():
loop = asyncio.get_event_loop()
keyring = kcache.KeyRing(keyring_path, loop)
controller = _init_controller(keyring, max_data_bytes)
reloader = _init_reloader(keyring_path, keyring)
LOG.info(f'Watching keyring directory {keyring_path}...')
reloader.start()
LOG.info('Starting the daemon...')
controller.start()
try:
loop.run_until_complete(_sleep())
keyring = kcache.init_keyring()
controller = _init_controller(keyring, max_data_bytes)
keyring.post_init_hook()
LOG.info('Starting the daemon with GnuPG=%s, socket=%s, database=%s',
keyring_path,
conf.daemon_params(),
conf.get_item('database', 'url'))
controller.start()
await _sleep()
except KeyboardInterrupt:
LOG.info("Finishing...")
except:
LOG.exception('Unexpected exception caught, your system may be unstable')
finally:
LOG.info('Shutting down keyring watcher and the daemon...')
reloader.stop()
reloader.join()
keyring.shutdown()
controller.stop()
LOG.info("Done")
if __name__ == '__main__':
_main()
asyncio.run(_main())

64
lacre/dbschema.py Normal file
View File

@ -0,0 +1,64 @@
"""Database schema for Lacre.
This definition includes:
- 'lacre_keys' -- temporary key storage, used by the frontend to submit keys and
by webgate-cron script to import submitted keys.
- 'lacre_identities' -- identity catalogue, used by encryption logic to match
emails with corresponding keys.
- 'lacre_locks' -- used only by the frontend.
"""
import sqlalchemy
# Values for lacre_keys.status column:
# - ST_DEFAULT: initial state;
# - ST_IMPORTED: key has been successfully processed by cron job;
# - ST_TO_BE_DELETED: key can be deleted.
ST_DEFAULT, ST_IMPORTED, ST_TO_BE_DELETED = range(3)
# lacre_keys.confirmed is set to an empty string when a key is confirmed by the user.
CO_CONFIRMED = ''
_meta = sqlalchemy.MetaData()
LACRE_KEYS = sqlalchemy.Table('lacre_keys', _meta,
sqlalchemy.Column('id', sqlalchemy.Integer, primary_key=True, nullable=False, autoincrement='auto'),
sqlalchemy.Column('email', sqlalchemy.String(256), index=True),
# ASCII-armored key
sqlalchemy.Column('publickey', sqlalchemy.Text),
# Empty string means this key has been confirmed.
sqlalchemy.Column('confirm', sqlalchemy.String(32)),
# Status: see ST_* constants at the top of the file.
sqlalchemy.Column('status', sqlalchemy.Integer, nullable=False, default=0),
sqlalchemy.Column('time', sqlalchemy.DateTime))
LACRE_LOCKS = sqlalchemy.Table('lacre_locks', _meta,
sqlalchemy.Column('id', sqlalchemy.Integer, primary_key=True, nullable=False, autoincrement='auto'),
sqlalchemy.Column('ip', sqlalchemy.String(16)),
sqlalchemy.Column('time', sqlalchemy.Integer),
sqlalchemy.Column('action', sqlalchemy.String(16)),
sqlalchemy.Column('num', sqlalchemy.Integer),
)
LACRE_IDENTITIES = sqlalchemy.Table('lacre_identities', _meta,
sqlalchemy.Column('email', sqlalchemy.String(256), index=True, nullable=False),
# Key fingerprint
sqlalchemy.Column('fingerprint', sqlalchemy.String(64), index=True, nullable=False))
def init_identities_table() -> sqlalchemy.Table:
return LACRE_IDENTITIES
def init_locks_table() -> sqlalchemy.Table:
return LACRE_LOCKS
def init_keys_table() -> sqlalchemy.Table:
return LACRE_KEYS
def create_tables(engine):
_meta.create_all(engine)
def table_metadata():
return _meta

View File

@ -4,172 +4,25 @@ IMPORTANT: This module has to be loaded _after_ initialisation of the logging
module.
"""
import lacre.text as text
import lacre.config as conf
from lacre._keyringcommon import KeyRing, KeyCache
from lacre.repositories import IdentityRepository, init_engine
import logging
from os import stat
from watchdog.events import FileSystemEventHandler, FileSystemEvent
from asyncio import Semaphore, create_task, get_event_loop, run
import copy
import GnuPG
LOG = logging.getLogger(__name__)
def _sanitize(keys):
sanitize = text.choose_sanitizer(conf.get_item('default', 'mail_case_insensitive'))
return {fingerprint: sanitize(keys[fingerprint]) for fingerprint in keys}
def init_keyring() -> KeyRing:
"""Initialise appropriate type of keyring."""
url = conf.get_item('database', 'url')
db_engine = init_engine(url)
return IdentityRepository(engine=db_engine)
class KeyCacheMisconfiguration(Exception):
"""Exception used to signal that KeyCache is misconfigured."""
class KeyCache:
"""A store for OpenPGP keys.
Key case is sanitised while loading from GnuPG if so
configured. See mail_case_insensitive parameter in section
[default].
"""
def __init__(self, keys: dict = None):
"""Initialise an empty cache.
With keyring_dir given, set location of the directory from which keys should be loaded.
"""
self._keys = keys
def __getitem__(self, fingerpring):
"""Look up email assigned to the given fingerprint."""
return self._keys[fingerpring]
def __setitem__(self, fingerprint, email):
"""Assign an email to a fingerpring, overwriting it if it was already present."""
self._keys[fingerprint] = email
def __contains__(self, fingerprint):
"""Check if the given fingerprint is assigned to an email."""
# This method has to be present for KeyCache to be a dict substitute.
# See mailgate, function _identify_gpg_recipients.
return fingerprint in self._keys
def has_email(self, email):
"""Check if cache contains a key assigned to the given email."""
return email in self._keys.values()
def __repr__(self):
"""Return text representation of this object."""
details = ' '.join(self._keys.keys())
return '<KeyCache %s>' % (details)
class KeyRing:
"""A high-level adapter for GnuPG-maintained keyring directory.
Its role is to keep a cache of keys present in the keyring,
reload it when necessary and produce static copies of
fingerprint=>email maps.
"""
def __init__(self, path: str, loop=None):
"""Initialise the adapter."""
self._path = path
self._keys = self._load_and_sanitize()
self._sema = Semaphore()
self._last_mod = None
self._loop = loop or get_event_loop()
def _load_and_sanitize(self):
keys = self._load_keyring_from(self._path)
return _sanitize(keys)
def _load_keyring_from(self, keyring_dir):
return GnuPG.public_keys(keyring_dir)
async def freeze_identities(self) -> KeyCache:
"""Return a static, async-safe copy of the identity map."""
async with self._sema:
keys = copy.deepcopy(self._keys)
return KeyCache(keys)
def load(self):
"""Load keyring, replacing any previous contents of the cache."""
LOG.debug('Reloading keys...')
tsk = create_task(self._load(), 'LoadTask')
self._loop.run_until_complete(tsk)
async def _load(self):
last_mod = self._read_mod_time()
LOG.debug(f'Keyring was last modified: {last_mod}')
if self._is_modified(last_mod):
LOG.debug('Keyring has been modified')
async with self._sema:
LOG.debug('About to re-load the keyring')
self.replace_keyring(self._load_keyring_from(self._path))
else:
LOG.debug('Keyring not modified recently, continuing')
self._last_mod = self._read_mod_time()
reload = load
def replace_keyring(self, keys: dict):
"""Overwrite previously stored key cache with KEYS."""
keys = _sanitize(keys)
LOG.info(f'Storing {len(keys)} keys')
self._keys = keys
def _read_mod_time(self) -> int:
# (mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime)
# 0 1 2 3 4 5 6 7 8 9
MTIME = 8
st = stat(self._path)
return st[MTIME]
def _is_modified(self, last_mod):
if self._last_mod is None:
LOG.debug('Keyring not loaded before')
return True
elif self._last_mod != last_mod:
LOG.debug('Keyring directory mtime changed')
return True
else:
LOG.debug('Keyring not modified ')
return False
def __repr__(self) -> str:
"""Return text representation of this keyring."""
return '<KeyRing path=%s last_mod=%d>' % (self._path, self._last_mod)
class KeyringModificationListener(FileSystemEventHandler):
"""A filesystem event listener that triggers key cache reload."""
def __init__(self, keyring: KeyRing):
"""Initialise a listener with a callback to be executed upon each change."""
self._keyring = keyring
def handle(self, event: FileSystemEvent):
"""Reload keys upon FS event."""
LOG.debug('FS event: %s, %s', event.event_type, event.src_path)
if 'pubring.kbx' in event.src_path:
LOG.info('Reloading %s on event: %s', self._keyring, event)
self._keyring.reload()
# All methods should do the same: reload the key cache.
# on_created = handle
# on_deleted = handle
on_modified = handle
def freeze_and_load_keys():
def freeze_and_load_keys() -> KeyCache:
"""Load and return keys.
Doesn't refresh the keys when they change on disk.
'"""
keyring_dir = conf.get_item('gpg', 'keyhome')
keyring = KeyRing(keyring_dir)
return run(keyring.freeze_identities())
"""
keyring = init_keyring()
return keyring.freeze_identities()

View File

@ -22,6 +22,11 @@ from email.policy import SMTP, SMTPUTF8
LOG = logging.getLogger(__name__)
class MailSerialisationException(BaseException):
"""We can't turn an EmailMessage into sequence of bytes."""
pass
class MailOperation:
"""Contract for an operation to be performed on a message."""
@ -124,7 +129,10 @@ class KeepIntact(MailOperation):
def perform(self, message: Message) -> bytes:
"""Return MESSAGE unmodified."""
return message.as_bytes(policy=SMTPUTF8)
try:
return message.as_bytes(policy=SMTPUTF8)
except IndexError as e:
raise MailSerialisationException(e)
def __repr__(self):
"""Return representation with just method and email."""

54
lacre/notify.py Normal file
View File

@ -0,0 +1,54 @@
"""Lacre notification sender"""
import logging
import lacre
import lacre.config as conf
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import markdown
LOG = logging.getLogger(__name__)
def _load_file(name):
f = open(name)
data = f.read()
f.close()
return data
def _authenticate_maybe(smtp):
if conf.config_item_equals('smtp', 'enabled', 'true'):
LOG.debug(f"Connecting to {conf.get_item('smtp', 'host')}:{conf.get_item('smtp', 'port')}")
smtp.connect(conf.get_item('smtp', 'host'), conf.get_item('smtp', 'port'))
smtp.ehlo()
if conf.config_item_equals('smtp', 'starttls', 'true'):
LOG.debug("StartTLS enabled")
smtp.starttls()
smtp.ehlo()
smtp.login(conf.get_item('smtp', 'username'), conf.get_item('smtp', 'password'))
def notify(mailsubject, messagefile, recipients = None):
"""Send notification email."""
mailbody = _load_file(conf.get_item('cron', 'mail_templates') + "/" + messagefile)
msg = MIMEMultipart("alternative")
msg["From"] = conf.get_item('cron', 'notification_email')
msg["To"] = recipients
msg["Subject"] = mailsubject
msg.attach(MIMEText(mailbody, 'plain'))
msg.attach(MIMEText(markdown.markdown(mailbody), 'html'))
if conf.config_item_set('relay', 'host') and conf.config_item_set('relay', 'enc_port'):
(host, port) = conf.relay_params()
smtp = smtplib.SMTP(host, port)
_authenticate_maybe(smtp)
LOG.info('Delivering notification: %s', recipients)
smtp.sendmail(conf.get_item('cron', 'notification_email'), recipients, msg.as_string())
else:
LOG.warning("Could not send mail due to wrong configuration")

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
"""Recipient processing package.
@ -53,13 +53,13 @@ class GpgRecipient(Recipient):
def __init__(self, left, right):
"""Initialise a tuple-like object that contains GPG recipient data."""
self._left = left
super().__init__(left)
self._right = right
def __getitem__(self, index):
"""Pretend this object is a tuple by returning an indexed tuple element."""
if index == 0:
return self._left
return self.email()
elif index == 1:
return self._right
else:
@ -67,11 +67,9 @@ class GpgRecipient(Recipient):
def __repr__(self):
"""Return textual representation of this GPG Recipient."""
return f"GpgRecipient({self._left!r}, {self._right!r})"
return f"GpgRecipient({self.email()!r}, {self._right!r})"
def email(self) -> str:
"""Return this recipient's email address."""
return self._left
__str__ = __repr__
def key(self):
"""Return this recipient's key ID."""

200
lacre/repositories.py Normal file
View File

@ -0,0 +1,200 @@
"""Lacre identity and key repositories."""
from sqlalchemy import create_engine, select, delete, and_, func
from sqlalchemy.exc import OperationalError
import logging
from lacre.config import flag_enabled, config_item_set, get_item, PoolingMode
from lacre._keyringcommon import KeyRing, KeyCache
import lacre.dbschema as db
LOG = logging.getLogger(__name__)
_HOUR_IN_SECONDS = 3600
# Internal state
_engine = None
def init_engine(url, db_debug=False):
global _engine
if not _engine:
config = _conn_config(db_debug)
_engine = create_engine(url, **config)
return _engine
def _conn_config(db_debug):
config = dict()
mode = PoolingMode.from_config('database', 'pooling_mode', required=True)
if mode is PoolingMode.OPTIMISTIC:
# Optimistic distonnect-handling: recycle connections.
config['pool_recycle'] = int(get_item('database', 'max_connection_age', _HOUR_IN_SECONDS))
elif mode is PoolingMode.PESSIMISTIC:
# Pessimistic disconnect-handling: pre_ping.
config['pool_pre_ping'] = True
# Additional pool settings
if config_item_set('database', 'pool_size'):
config['pool_size'] = int(get_item('database', 'pool_size'))
if config_item_set('database', 'max_overflow'):
config['max_overflow'] = int(get_item('database', 'max_overflow'))
if db_debug:
config['echo'] = 'debug'
config['echo_pool'] = 'debug'
LOG.debug('Database engine configuration: %s', config)
return config
class IdentityRepository(KeyRing):
def __init__(self, /, connection=None, *, engine):
self._identities = db.LACRE_IDENTITIES
self._engine = engine
def register_or_update(self, email, fprint):
assert email, "email is mandatory"
assert fprint, "fprint is mandatory"
if self._exists(email):
self._update(email, fprint)
else:
self._insert(email, fprint)
def _exists(self, email: str) -> bool:
selq = select(self._identities.c.email).where(self._identities.c.email == email)
with self._engine.connect() as conn:
return [e for e in conn.execute(selq)]
def _insert(self, email, fprint):
insq = self._identities.insert().values(email=email, fingerprint=fprint)
LOG.debug('Registering identity: %s -- %s', insq, insq.compile().params)
with self._engine.connect() as conn:
conn.execute(insq)
def _update(self, email, fprint):
upq = self._identities.update() \
.values(fingerprint=fprint) \
.where(self._identities.c.email == email)
LOG.debug('Updating identity: %s -- %s', upq, upq.compile().params)
with self._engine.connect() as conn:
conn.execute(upq)
def delete(self, email):
delq = delete(self._identities).where(self._identities.c.email == email)
LOG.debug('Deleting assigned keys: %s -- %s', delq, delq.compile().params)
with self._engine.connect() as conn:
conn.execute(delq)
def delete_all(self):
LOG.warn('Deleting all identities from the database')
delq = delete(self._identities)
with self._engine.connect() as conn:
conn.execute(delq)
def freeze_identities(self) -> KeyCache:
"""Return a static, async-safe copy of the identity map.
Depending on the value of [daemon]bounce_on_keys_missing value,
if we get a database exception, this method will either return
empty collection or let the exception be propagated.
"""
try:
return self._load_identities()
except OperationalError:
if flag_enabled('daemon', 'bounce_on_keys_missing'):
raise
else:
LOG.exception('Failed to load keys, returning empty collection')
return KeyCache({})
def _load_identities(self) -> KeyCache:
all_identities = select(self._identities.c.fingerprint, self._identities.c.email)
with self._engine.connect() as conn:
result = conn.execute(all_identities)
LOG.debug('Retrieving all keys: %s', all_identities)
return KeyCache({key_id: email for key_id, email in result})
class KeyConfirmationQueue:
"""Encapsulates access to lacre_keys table."""
# Default number of items retrieved from the database.
keys_read_max = 100
def __init__(self, /, engine):
self._keys = db.LACRE_KEYS
self._engine = engine
def fetch_keys(self, /, max_keys=None):
"""Runs a query to retrieve at most `keys_read_max` keys and returns db result."""
max_keys = max_keys or self.keys_read_max
LOG.debug('Row limit: %d', max_keys)
selq = select(self._keys.c.publickey, self._keys.c.id, self._keys.c.email) \
.where(and_(self._keys.c.status == db.ST_DEFAULT, self._keys.c.confirm == db.CO_CONFIRMED)) \
.limit(max_keys)
LOG.debug('Retrieving keys to be processed: %s -- %s', selq, selq.compile().params)
with self._engine.connect() as conn:
return [e for e in conn.execute(selq)]
def count_keys(self):
selq = select(func.count(self._keys.c.id)) \
.where(and_(self._keys.c.status == db.ST_DEFAULT, self._keys.c.confirm == db.CO_CONFIRMED))
LOG.debug('Counting all keys: %s -- %s', selq, selq.compile().params)
try:
with self._engine.connect() as conn:
res = conn.execute(selq)
# This is a 1-element tuple.
return res.one_or_none()[0]
except OperationalError:
LOG.exception('Cannot count keys')
return None
def fetch_keys_to_delete(self):
seldel = select(self._keys.c.email, self._keys.c.id) \
.where(self._keys.c.status == db.ST_TO_BE_DELETED) \
.limit(self.keys_read_max)
with self._engine.connect() as conn:
return [e for e in conn.execute(seldel)]
def delete_keys(self, row_id, /, email=None):
"""Remove key from the database."""
if email is not None:
LOG.debug('Deleting key: id=%s, email=%s', row_id, email)
delq = delete(self._keys).where(and_(self._keys.c.email == email, self._keys.c.id == row_id))
else:
LOG.debug('Deleting key: id=%s', row_id)
delq = delete(self._keys).where(self._keys.c.id == row_id)
with self._engine.connect() as conn:
LOG.debug('Deleting public keys associated with confirmed email: %s', delq)
conn.execute(delq)
def delete_key_by_email(self, email):
"""Remove keys linked to the given email from the database."""
delq = delete(self._keys).where(self._keys.c.email == email)
LOG.debug('Deleting email for: %s', email)
with self._engine.connect() as conn:
conn.execute(delq)
def mark_accepted(self, row_id):
modq = self._keys.update().where(self._keys.c.id == row_id).values(status=db.ST_IMPORTED)
LOG.debug("Key imported, updating key: %s", modq)
with self._engine.connect() as conn:
conn.execute(modq)

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
"""S/MIME handling module."""
@ -75,7 +75,7 @@ def encrypt(raw_message, recipients, from_addr):
out.write('Subject: ' + raw_message['Subject'] + text.EOL_S)
if conf.config_item_equals('default', 'add_header', 'yes'):
out.write('X-GPG-Mailgate: Encrypted by GPG Mailgate' + text.EOL_S)
out.write('X-Lacre: Encrypted by Lacre' + text.EOL_S)
s.write(out, p7)

29
lacre/stats.py Normal file
View File

@ -0,0 +1,29 @@
"""Insights into Lacre's inner workings."""
import time
import logging
class ExecutionTimeLogger:
"""Context-manager that measures how much time some operation took and logs it."""
def __init__(self, message: str, logger: logging.Logger):
self._message = message
self._log = logger
self._start = None
def __enter__(self):
self._start = time.process_time()
self._log.info('Start: %s', self._message)
def __exit__(self, exc_type=None, exc_value=None, traceback=None):
end = time.process_time()
ellapsed = (end - self._start) * 1000
if exc_type:
exception = (exc_type, exc_value, traceback)
self._log.error('%s took %d ms, raised exception', self._message, ellapsed, exc_info=exception)
else:
self._log.info('%s took %d ms', self._message, ellapsed)
def time_logger(msg: str, logger: logging.Logger):
return ExecutionTimeLogger(msg, logger)

View File

@ -3,4 +3,3 @@ SQLAlchemy==1.4.32
Markdown==3.4.1
M2Crypto==0.38.0
requests==2.27.1
watchdog==2.1.9

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
import configparser
@ -30,7 +30,7 @@ def _spawn(cmd):
"PATH": os.getenv("PATH"),
"PYTHONPATH": os.getcwd(),
"LANG": 'en_US.UTF-8',
"GPG_MAILGATE_CONFIG": "test/gpg-mailgate-daemon-test.conf"
"LACRE_CONFIG": "test/lacre-daemon.conf"
}
logging.debug(f"Spawning command: {cmd} with environment: {env_dict!r}")
return subprocess.Popen(cmd,

View File

@ -32,8 +32,8 @@ certs: test/certs
e2e_log: test/logs/e2e.log
e2e_log_format: %(asctime)s %(pathname)s:%(lineno)d %(levelname)s [%(funcName)s] %(message)s
e2e_log_datefmt: %Y-%m-%d %H:%M:%S
lacre_log: test/logs/gpg-mailgate.log
log_config: test/gpg-lacre-log.ini
lacre_log: test/logs/lacre-simple.log
log_config: test/lacre-logging.conf
# TEST IDENTITIES AND SETTINGS:
#

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
import os
@ -27,7 +27,7 @@ import unittest
RELAY_SCRIPT = "test/utils/relay.py"
CONFIG_FILE = "test/gpg-mailgate.conf"
CONFIG_FILE = "test/lacre.conf"
def _build_config(config):
@ -39,6 +39,11 @@ def _build_config(config):
cp.add_section("gpg")
cp.set("gpg", "keyhome", config["gpg_keyhome"])
cp.add_section('database')
cp.set('database', 'enabled', 'yes')
cp.set('database', 'url', 'sqlite:///test/lacre.db')
cp.set('database', 'pooling_mode', 'optimistic')
cp.add_section("smime")
cp.set("smime", "cert_path", config["smime_certpath"])
@ -124,7 +129,7 @@ class SimpleMailFilterE2ETest(unittest.TestCase):
following properties: 'descr', 'to', 'in' and 'out'.
"""
gpglacre_cmd = self._python_command(
'gpg-mailgate.py',
'lacre.py',
self._e2e_config.get(case_name, 'to'))
relay_cmd = self._python_command(
@ -142,7 +147,7 @@ class SimpleMailFilterE2ETest(unittest.TestCase):
gpglacre_proc = subprocess.run(gpglacre_cmd,
input=_load_file(self._e2e_config.get(case_name, "in")),
capture_output=True,
env={"GPG_MAILGATE_CONFIG": self._e2e_config_path,
env={"LACRE_CONFIG": self._e2e_config_path,
"PATH": os.getenv("PATH")})
# Let the relay process the data.

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@ -1,5 +1,5 @@
[logging]
config = test/gpg-lacre-log.ini
config = test/lacre-logging.conf
file = test/logs/gpg-mailgate.log
format = %(asctime)s %(module)s[%(process)d]: %(message)s
date_format = ISO
@ -10,9 +10,15 @@ keyhome = test/keyhome
[smime]
cert_path = test/certs
[daemon]
host = not_used
port = not_used
[database]
enabled = yes
url = sqlite:///test/lacre.db
pooling_mode = optimistic
max_connection_age = 3600
[relay]
host = localhost
@ -20,6 +26,7 @@ port = 2500
[cron]
send_email = no
mail_templates = not_used
[enc_keymap]
alice@disposlab = 1CD245308F0963D038E88357973CF4D9387C44D7

View File

@ -1,12 +1,11 @@
[logging]
config = test/gpg-lacre-log.ini
config = test/lacre-logging.conf
file = test/logs/gpg-mailgate.log
format = %(asctime)s %(module)s[%(process)d]: %(message)s
date_format = ISO
[gpg]
keyhome = test/keyhome
cache_refresh_minutes = 1
[smime]
cert_path = test/certs
@ -14,6 +13,7 @@ cert_path = test/certs
[database]
enabled = yes
url = sqlite:///test/lacre.db
pooling_mode = optimistic
[relay]
host = localhost
@ -26,6 +26,7 @@ log_headers = yes
[cron]
send_email = no
mail_templates = not_used
[pgp_style]
# this recipient has PGP/MIME enabled, because the default approach is to use

View File

@ -1,20 +1,20 @@
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
"""Unit-tests as contracts for external dependencies.
@ -27,7 +27,8 @@ documentation.
import email
import email.mime.multipart
from email.message import EmailMessage
from email.policy import SMTP
from email.policy import SMTP, SMTPUTF8
from email.errors import HeaderParseError
import unittest
from configparser import RawConfigParser
@ -165,6 +166,26 @@ class EmailParsingTest(unittest.TestCase):
self.assertIsInstance(payload, str)
self.assertTrue(message_boundary in payload)
def test_fail_if_message_id_parsing_is_fixed(self):
# Unfortunately, Microsoft sends messages with Message-Id header values
# that email parser can't process.
#
# Bug: https://github.com/python/cpython/issues/105802
# Fix: https://github.com/python/cpython/pull/108133
rawmsg = b"From: alice@lacre.io\r\n" \
+ b"To: bob@lacre.io\r\n" \
+ b"Subject: Test message\r\n" \
+ b"Content-Type: text/plain\r\n" \
+ b"Content-Transfer-Encoding: base64\r\n" \
+ b"Message-Id: <[yada-yada-yada@microsoft.com]>\r\n" \
+ b"\r\n" \
+ b"SGVsbG8sIFdvcmxkIQo=\r\n"
msg = email.message_from_bytes(rawmsg, policy=SMTPUTF8)
self.assertEqual(len(msg.defects), 0)
self.assertRaises(IndexError, lambda: msg['Message-Id'])
class EmailTest(unittest.TestCase):
def test_boundary_generated_after_as_string_call(self):

View File

@ -1,8 +1,14 @@
import GnuPG
import logging
import unittest
class GnuPGUtilitiesTest(unittest.TestCase):
def setUp(self):
# Record GnuPG logs:
logging.basicConfig(filename='test/logs/unittest.log', level=logging.DEBUG,
format='%(asctime)s %(pathname)s:%(lineno)d %(levelname)s [%(funcName)s] %(message)s')
def test_build_default_command(self):
cmd = GnuPG._build_command("test/keyhome")
self.assertEqual(cmd, ["gpg", "--homedir", "test/keyhome"])
@ -37,18 +43,28 @@ class GnuPGUtilitiesTest(unittest.TestCase):
self.assertDictEqual(keys, known_identities)
def test_add_delete_key(self):
self.assertDictEqual(GnuPG.public_keys('/tmp'), { })
GnuPG.add_key('/tmp', self._load('test/keys/bob@disposlab.pub'))
self.assertDictEqual(GnuPG.public_keys('/tmp'), {
self.assertDictEqual(GnuPG.public_keys('test/keyhome.other'), { })
GnuPG.add_key('test/keyhome.other', self._load('test/keys/bob@disposlab.pub'))
self.assertDictEqual(GnuPG.public_keys('test/keyhome.other'), {
'19CF4B47ECC9C47AFA84D4BD96F39FDA0E31BB67': 'bob@disposlab',
})
GnuPG.delete_key('/tmp', 'bob@disposlab')
self.assertDictEqual(GnuPG.public_keys('/tmp'), { })
GnuPG.delete_key('test/keyhome.other', 'bob@disposlab')
self.assertDictEqual(GnuPG.public_keys('test/keyhome.other'), { })
def _load(self, filename):
with open(filename) as f:
return f.read()
def test_extract_fingerprint(self):
sample_in = '''fpr:::::::::1CD245308F0963D038E88357973CF4D9387C44D7:'''
fpr = GnuPG._extract_fingerprint(sample_in)
self.assertEqual(fpr, '1CD245308F0963D038E88357973CF4D9387C44D7')
def test_parse_uid_line(self):
sample_in = '''uid:e::::1624794010::C16E259AA1435947C6385B8160BC020B6C05EE18::alice@disposlab::::::::::0:'''
uid = GnuPG._parse_uid_line(sample_in)
self.assertEqual(uid, 'alice@disposlab')
def test_parse_statusfd_key_expired(self):
key_expired = b"""
[GNUPG:] KEYEXPIRED 1668272263
@ -56,10 +72,24 @@ class GnuPGUtilitiesTest(unittest.TestCase):
[GNUPG:] INV_RECP 0 name@domain
[GNUPG:] FAILURE encrypt 1
"""
result = GnuPG.parse_status(key_expired)
self.assertEqual(result['issue'], b'KEYEXPIRED')
self.assertEqual(result['issue'], 'key expired')
self.assertEqual(result['recipient'], b'name@domain')
self.assertEqual(result['cause'], 'No specific reason given')
self.assertEqual(result['cause'], 'Unknown')
self.assertEqual(result['key'], b'XXXXXXXXXXXXX')
def test_parse_statusfd_key_absent(self):
non_specific_errors = b"""
[GNUPG:] INV_RECP 0 name@domain
[GNUPG:] FAILURE encrypt 1
"""
result = GnuPG.parse_status(non_specific_errors)
self.assertEqual(result['issue'], b'n/a')
self.assertEqual(result['recipient'], b'name@domain')
self.assertEqual(result['cause'], 'Unknown')
self.assertEqual(result['key'], b'n/a')
if __name__ == '__main__':

View File

@ -40,3 +40,17 @@ class LacreCoreTest(unittest.TestCase):
'only content and content-type should be copied')
self.assertEqual(rewrapped.get_content_type(), 'text/plain',
'rewrapped part should have initial message\'s content-type')
def test_payload_wrapping_wo_content_type(self):
m = EmailMessage()
m.set_payload('This is a payload.\r\n'
+ '\r\n'
+ 'It has two paragraphs.\r\n')
m['Subject'] = 'Source message'
rewrapped = lacre.core._rewrap_payload(m)
self.assertFalse('Subject' in rewrapped,
'only content and content-type should be copied')
self.assertEqual(rewrapped.get_content_type(), 'text/plain',
'rewrapped part should have initial message\'s content-type')

View File

@ -0,0 +1,24 @@
"""Lacre identity and key repository tests."""
import unittest
import lacre.config as conf
import lacre.repositories as r
import lacre.dbschema as s
def ignore_sql(sql, *args, **kwargs):
pass
class IdentityRepositoryTest(unittest.TestCase):
def setUpClass():
# required for init_engine to work
conf.load_config()
def test_freeze_identities(self):
eng = r.init_engine('sqlite:///test/lacre.db')
ir = r.IdentityRepository(engine=eng)
identities = ir.freeze_identities()
self.assertTrue(identities)

View File

@ -0,0 +1,38 @@
import unittest
from logging import getLogger, ERROR, Handler
from lacre.stats import time_logger
def make_exception_raiser(logger):
def f():
with time_logger('Just a test', logger):
logger.info('Doing something')
raise Exception('this is a test')
return f
class LogRecordCollector(Handler):
logged_records = []
def handle(self, r):
self.logged_records.append(self.format(r))
class ExecutionTimeLoggerTest(unittest.TestCase):
def test_exception_handling(self):
handler = LogRecordCollector()
logger = getLogger('test-logger')
logger.addHandler(handler)
f = make_exception_raiser(logger)
self.assertRaises(Exception, f)
self.assertLogs(logger, ERROR)
self.assertEqual(len(handler.logged_records), 3)
self.assertEqual(handler.logged_records[0], 'Start: Just a test')
self.assertEqual(handler.logged_records[1], 'Doing something')
# Exception record should include the timing result and the traceback...
self.assertRegex(handler.logged_records[2], '^Just a test took \\d ms, raised exception\nTraceback.*')
# ...as well as the original exception
self.assertRegex(handler.logged_records[2], 'Exception: this is a test$')

View File

@ -1,7 +1,7 @@
#!/usr/local/bin/python2
#
# This quick-and-dirty script supports only the happy case of SMTP session,
# i.e. what gpg-mailgate/gpg-lacre needs to deliver encrypted email.
# i.e. what lacre needs to deliver encrypted email.
#
# It listens on the port given as the only command-line argument and consumes a
# message, then prints it to standard output. The goal is to be able to

View File

@ -3,23 +3,27 @@ import sqlalchemy
from sqlalchemy.sql import insert
def define_db_schema():
meta = sqlalchemy.MetaData()
meta = sqlalchemy.MetaData()
gpgmw_keys = sqlalchemy.Table('gpgmw_keys', meta,
sqlalchemy.Column('id', sqlalchemy.Integer, primary_key=True),
sqlalchemy.Column('email', sqlalchemy.String(256)),
sqlalchemy.Column('publickey', sqlalchemy.Text),
sqlalchemy.Column('confirm', sqlalchemy.String(32)),
sqlalchemy.Column('status', sqlalchemy.Integer),
sqlalchemy.Column('time', sqlalchemy.DateTime))
lacre_keys = sqlalchemy.Table('lacre_keys', meta,
sqlalchemy.Column('id', sqlalchemy.Integer, primary_key=True),
sqlalchemy.Column('email', sqlalchemy.String(256)),
sqlalchemy.Column('publickey', sqlalchemy.Text),
sqlalchemy.Column('confirm', sqlalchemy.String(32)),
sqlalchemy.Column('status', sqlalchemy.Integer),
sqlalchemy.Column('time', sqlalchemy.DateTime))
return (meta, gpgmw_keys)
identities = sqlalchemy.Table('lacre_identities', meta,
sqlalchemy.Column('email', sqlalchemy.String(256), index=True),
sqlalchemy.Column('fingerprint', sqlalchemy.String(64), index=True))
return (meta, lacre_keys, identities)
if len(sys.argv) != 2:
print("ERROR: output database missing")
sys.exit(1)
print("ERROR: output database missing")
sys.exit(1)
(meta, gpgmw_keys) = define_db_schema()
(meta, lacre_keys, identities) = define_db_schema()
dbname = sys.argv[1]
test_db = sqlalchemy.create_engine(f"sqlite:///{dbname}")
@ -30,8 +34,8 @@ meta.create_all(test_db)
conn = test_db.connect()
# Populate the database with dummy data
conn.execute(gpgmw_keys.insert(), [
{"id": 1, "email": "alice@disposlab", "publickey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\
conn.execute(lacre_keys.insert(), [
{"id": 1, "email": "alice@disposlab", "publickey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\
\n\
mQGNBGDYY5oBDAC+HAVjA05jsIpHfQ2KQ9m2olo1Qnlk+dkjD+Gagxj1ACezyiGL\n\
cfZfoE/MJYLCH9yPcX1fUIAPwdAyfJKlvkVcz+MhEpgl3aP3NM2L2unSx3v9ZFwT\n\
@ -73,7 +77,7 @@ pw==\n\
=Tbwz\n\
-----END PGP PUBLIC KEY BLOCK-----\
", "status": 0, "confirm": "", "time": None},
{"id": 2, "email": "bob@disposlab", "publickey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\
{"id": 2, "email": "bob@disposlab", "publickey": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\
\n\
mDMEYdTFkRYJKwYBBAHaRw8BAQdA2tgdP1pMt3cv3XAW7ov5AFn74mMZvyTksp9Q\n\
eO1PkpK0GkJvYiBGb29iYXIgPGJvYkBkaXNwb3NsYWI+iJYEExYIAD4WIQQZz0tH\n\
@ -86,6 +90,12 @@ AQgHiHgEGBYIACAWIQQZz0tH7MnEevqE1L2W85/aDjG7ZwUCYdTF4AIbDAAKCRCW\n\
OjjB6xRD0Q2FN+alsNGCtdutAs18AZ5l33RMzws=\n\
=wWoq\n\
-----END PGP PUBLIC KEY BLOCK-----\
", "status": 0, "confirm": "", "time": None},
{"id": 3, "email": "cecil@lacre.io", "publickey": "RUBBISH", "status": 0, "confirm": "", "time": None}
])
", "status": 1, "confirm": "", "time": None},
{"id": 3, "email": "cecil@lacre.io", "publickey": "RUBBISH", "status": 2, "confirm": "", "time": None}
])
conn.execute(identities.insert(), [
{'fingerprint': '1CD245308F0963D038E88357973CF4D9387C44D7', 'email': 'alice@disposlab'},
{'fingerprint': '19CF4B47ECC9C47AFA84D4BD96F39FDA0E31BB67', 'email': 'bob@disposlab'},
{'fingerprint': '530B1BB2D0CC7971648198BBA4774E507D3AF5BC', 'email': 'evan@disposlab'}
])

View File

@ -1,154 +1,124 @@
#!/usr/bin/python
#
# gpg-mailgate
# lacre
#
# This file is part of the gpg-mailgate source code.
# This file is part of the lacre source code.
#
# gpg-mailgate is free software: you can redistribute it and/or modify
# lacre is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# gpg-mailgate source code is distributed in the hope that it will be useful,
# lacre source code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with gpg-mailgate source code. If not, see <http://www.gnu.org/licenses/>.
# along with lacre source code. If not, see <http://www.gnu.org/licenses/>.
#
import GnuPG
import sqlalchemy
from sqlalchemy.sql import select, delete, and_
import smtplib
import markdown
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import sys
import logging
import lacre
import lacre.config as conf
from lacre.notify import notify
def _load_file(name):
f = open(name)
data = f.read()
f.close()
return data
def _authenticate_maybe(smtp):
if conf.config_item_equals('smtp', 'enabled', 'true'):
LOG.debug(f"Connecting to {conf.get_item('smtp', 'host')}:{conf.get_item('smtp', 'port')}")
smtp.connect(conf.get_item('smtp', 'host'), conf.get_item('smtp', 'port'))
smtp.ehlo()
if conf.config_item_equals('smtp', 'starttls', 'true'):
LOG.debug("StartTLS enabled")
smtp.starttls()
smtp.ehlo()
smtp.login(conf.get_item('smtp', 'username'), conf.get_item('smtp', 'password'))
def _send_msg(mailsubject, messagefile, recipients = None):
mailbody = _load_file(conf.get_item('cron', 'mail_templates') + "/" + messagefile)
msg = MIMEMultipart("alternative")
msg["From"] = conf.get_item('cron', 'notification_email')
msg["To"] = recipients
msg["Subject"] = mailsubject
msg.attach(MIMEText(mailbody, 'plain'))
msg.attach(MIMEText(markdown.markdown(mailbody), 'html'))
if conf.config_item_set('relay', 'host') and conf.config_item_set('relay', 'enc_port'):
relay = (conf.get_item('relay', 'host'), int(conf.get_item('relay', 'enc_port')))
smtp = smtplib.SMTP(relay[0], relay[1])
_authenticate_maybe(smtp)
smtp.sendmail(conf.get_item('cron', 'notification_email'), recipients, msg.as_string())
else:
LOG.info("Could not send mail due to wrong configuration")
def _setup_db_connection(url):
engine = sqlalchemy.create_engine(url)
return (engine, engine.connect())
def _define_db_schema():
meta = sqlalchemy.MetaData()
gpgmw_keys = sqlalchemy.Table('gpgmw_keys', meta,
sqlalchemy.Column('id', sqlalchemy.Integer, primary_key=True),
sqlalchemy.Column('email', sqlalchemy.String(256)),
sqlalchemy.Column('publickey', sqlalchemy.Text),
sqlalchemy.Column('confirm', sqlalchemy.String(32)),
sqlalchemy.Column('status', sqlalchemy.Integer),
sqlalchemy.Column('time', sqlalchemy.DateTime))
return (gpgmw_keys)
# Read configuration from /etc/gpg-mailgate.conf
# Read configuration from /etc/lacre.conf
conf.load_config()
lacre.init_logging(conf.get_item('logging', 'config'))
LOG = logging.getLogger('webgate-cron.py')
import GnuPG
from lacre.repositories import KeyConfirmationQueue, IdentityRepository, init_engine
if conf.config_item_equals('database', 'enabled', 'yes') and conf.config_item_set('database', 'url'):
(engine, conn) = _setup_db_connection(conf.get_item("database", "url"))
(gpgmw_keys) = _define_db_schema()
selq = select(gpgmw_keys.c.publickey, gpgmw_keys.c.id, gpgmw_keys.c.email)\
.where(and_(gpgmw_keys.c.status == 0, gpgmw_keys.c.confirm == ""))\
.limit(100)
LOG.debug(f"Retrieving keys to be processed: {selq}")
result_set = conn.execute(selq)
def _validate_config():
missing = conf.validate_config(additional=conf.CRON_REQUIRED)
if missing:
LOG.error('Missing config parameters: %s', missing)
exit(lacre.EX_CONFIG)
for key_id, row_id, email in result_set:
# delete any other public keys associated with this confirmed email address
delq = delete(gpgmw_keys).where(and_(gpgmw_keys.c.email == email, gpgmw_keys.c.id != row_id))
LOG.debug(f"Deleting public keys associated with confirmed email: {delq}")
conn.execute(delq)
GnuPG.delete_key(conf.get_item('gpg', 'keyhome'), email)
LOG.info('Deleted key for <' + email + '> via import request')
if key_id.strip(): # we have this so that user can submit blank key to remove any encryption
if GnuPG.confirm_key(key_id, email):
GnuPG.add_key(conf.get_item('gpg', 'keyhome'), key_id) # import the key to gpg
modq = gpgmw_keys.update().where(gpgmw_keys.c.id == row_id).values(status=1)
LOG.debug(f"Key imported, updating key: {modq}")
conn.execute(modq) # mark key as accepted
LOG.warning('Imported key from <' + email + '>')
if conf.config_item_equals('cron', 'send_email', 'yes'):
_send_msg("PGP key registration successful", "registrationSuccess.md", email)
else:
delq = delete(gpgmw_keys).where(gpgmw_keys.c.id == row_id)
LOG.debug(f"Cannot confirm key, deleting it: {delq}")
conn.execute(delq) # delete key
LOG.warning('Import confirmation failed for <' + email + '>')
if conf.config_item_equals('cron', 'send_email', 'yes'):
_send_msg("PGP key registration failed", "registrationError.md", email)
else:
# delete key so we don't continue processing it
delq = delete(gpgmw_keys).where(gpgmw_keys.c.id == row_id)
LOG.debug(f"Deleting key: {delq}")
conn.execute(delq)
if conf.config_item_equals('cron', 'send_email', 'yes'):
_send_msg("PGP key deleted", "keyDeleted.md", email)
def import_key(key_dir, armored_key, key_id, email, key_queue, identities):
# import the key to gpg
(fingerprint, _) = GnuPG.add_key(key_dir, armored_key)
# delete keys
stat2q = select(gpgmw_keys.c.email, gpgmw_keys.c.id).where(gpgmw_keys.c.status == 2).limit(100)
stat2_result_set = conn.execute(stat2q)
key_queue.mark_accepted(key_id)
identities.register_or_update(email, fingerprint)
for email, row_id in stat2_result_set:
GnuPG.delete_key(conf.get_item('gpg', 'keyhome'), email)
delq = delete(gpgmw_keys).where(gpgmw_keys.c.id == row_id)
LOG.debug(f"Deleting keys that have already been processed: {delq}")
conn.execute(delq)
LOG.info('Deleted key for <' + email + '>')
else:
LOG.info('Imported key from: %s', email)
if conf.flag_enabled('cron', 'send_email'):
notify("PGP key registration successful", "registrationSuccess.md", email)
def import_failed(key_id, email, key_queue):
key_queue.delete_keys(key_id)
LOG.warning('Import confirmation failed: %s', email)
if conf.flag_enabled('cron', 'send_email'):
notify("PGP key registration failed", "registrationError.md", email)
def delete_key(key_id, email, key_queue):
# delete key so we don't continue processing it
LOG.debug('Empty key received, just deleting')
key_queue.delete_keys(row_id)
if conf.flag_enabled('cron', 'send_email'):
notify("PGP key deleted", "keyDeleted.md", email)
def cleanup(key_dir, key_queue):
"""Delete keys and queue entries."""
LOG.info('Cleaning up after a round of key confirmation')
for email, row_id in key_queue.fetch_keys_to_delete():
LOG.debug('Removing key from keyring: %s', email)
GnuPG.delete_key(key_dir, email)
LOG.debug('Removing key from identity store: %s', row_id)
key_queue.delete_keys(row_id)
LOG.info('Deleted key for: %s', email)
_validate_config()
if not (conf.flag_enabled('database', 'enabled') and conf.config_item_set('database', 'url')):
print("Warning: doing nothing since database settings are not configured!")
LOG.error("Warning: doing nothing since database settings are not configured!")
sys.exit(lacre.EX_CONFIG)
try:
db_engine = init_engine(conf.get_item('database', 'url'))
identities = IdentityRepository(engine=db_engine)
key_queue = KeyConfirmationQueue(engine=db_engine)
key_dir = conf.get_item('gpg', 'keyhome')
LOG.debug('Using GnuPG with home directory in %s', key_dir)
for armored_key, row_id, email in key_queue.fetch_keys():
# delete any other public keys associated with this confirmed email address
key_queue.delete_keys(row_id, email=email)
identities.delete(email)
GnuPG.delete_key(key_dir, email)
LOG.info('Deleted key via import request for: %s', email)
if not armored_key.strip(): # we have this so that user can submit blank key to remove any encryption
# delete key so we don't continue processing it
delete_key(row_id, email, key_queue)
continue
if GnuPG.confirm_key(armored_key, email):
import_key(key_dir, armored_key, row_id, email, key_queue, identities)
else:
import_failed(row_id, email, key_queue)
cleanup(key_dir, key_queue)
except:
LOG.exception('Unexpected issue during key confirmation')