This is a bug fix release. Fixes include:
* GPGME S/MIME non-detached signature handling.
* A compilation issue with ncurses-6.1 when tinfow is split out.
Hopefully the fix doesn't impact anyone: please let me know of any
issues.
* Regular expression crashes and weird behavior problems on MacOS (and
possibly other BSDs). This was most noticeable with the recent Xcode
9.3 release, but the issue has existed for a while.
* GSSAPI authentication issues, which may have affected Exchange users.
We've changed behavior to match the RFC. If you are negatively
impacted by this, please let me know.
- Changed the filenames of response record files so they sort by time in
a directory listing. This may cause extra responses after upgrading.
- Added support for putting the original sender (%s) and recipient (%r)
in the response message.
- Added support for single file config mode, including the response.
- Fixed handling of MySQL default timestamp value.
- [CritFix] Plug bad memory leak in protocol reply
- [Feature] Add avx2 codec for base64
- [Feature] Add method to receive all URL flags from Lua API
- [Feature] Allow to fold headers on stop characters
- [Feature] Allow to set lua_cpath from options
- [Feature] Allow to specify custom rejection message in milter
- [Feature] Deal with unnormalised Unicode obfuscation
- [Feature] Do not detect language twice for relative parts
- [Feature] Implement oversigning feature
- [Feature] Implement silent logging level to minimize noise in logs
- [Feature] Improve URL_IN_SUBJECT rule
- [Feature] Use hashing to reduce redis attack surface
- [Fix] Add oversigning for the most important headers
- [Fix] add 'rewrite subject' to History dropdown
- [Fix] Another fix in folding algorithm
- [Fix] Do not call multimap addr for parts of addr if filter is
presented
- [Fix] Do not clean hostname on generic reset
- [Fix] Do not create pid file in no-fork mode
- [Fix] Fix fold_after case to preserve multiple spaces
- [Fix] Fix folding and folding tests
- [Fix] Fix hostname usage in milter mode
- [Fix] Fix lua RSA verify and its tests
- [Fix] Fix metadata exporter send_mail backend (#2124)
- [Fix] Fix processing of '\v' in libucl
- [Fix] Fix shemaless URLs detection
- [Fix] Fix support of multiple headers in sign_header
- [Fix] Fix usage of util.parse_mail_address
- [Fix] Fix weights of dynamic squeezed rules
- [Fix] Leak from bucket before checking the burst
- [Fix] Stop using own localtime as DST could be messy in many cases
- [Fix] Treat unnormalised URLs as obscured
- [Rework] Restore leaky bucket model in ratelimit plugin
- [WebUI] Add messages total to throughput summary
- [WebUI] Add symbols order selector to history
- [WebUI] Config: Load list on demand
- [WebUI] Fix modalBody for maps that appear more than once
- [WebUI] History: Fix Tooltips on paging, filtering and sorting
- [WebUI] Remove a previously-attached event handler
- [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1
Changelog:
Fixed Searching message bodies of messages in local folders,
including filter and quick filter operations, did not find
content in message attachments
Fixed Better error handling for Yahoo accounts
Fixed Various security fixes
#CVE-2018-5127: Buffer overflow manipulating SVG animatedPathSegList
#CVE-2018-5129: Out-of-bounds write with malformed IPC messages
#CVE-2018-5144: Integer overflow during Unicode conversion
#CVE-2018-5146: Out of bounds memory write in libvorbis
#CVE-2018-5125: Memory safety bugs fixed in Firefox 59, Firefox ESR 52.7,
and Thunderbird 52.7
#CVE-2018-5145: Memory safety bugs fixed in Firefox ESR 52.7 and
Thunderbird 52.7
- [Feature] Store emails in Clickhouse
- [Feature] Support single quotes in config
- [Feature] Use templates when publishing CH schema
- [Feature] Improve Docker image
- [Fix] Add rounding when printing a lot of FP variables
- [Fix] Allow to disable certain actions by assigning null to them
- [Fix] Disable results caching
- [Fix] Fix disabling of squeezed symbols
- [Fix] Fix scan time set
- [Fix] Rework logic of actions setting
- [Fix] Try to fix various Lua stack issues
- [WebUI] Add link tag for favicon.ico
- [WebUI] Display hostname:port/path in the page title
- [CritFix] Fix lowercase comparison
- [CritFix] Timezone defines seconds WEST UTC not East
- [Feature] Add filename to log format
- [Feature] Add lua rules squeezing
- [Feature] Add related symbols analysis to rspamd_stats
- [Feature] Remove upstream `X-Spam: Yes` header by default
- [Feature] rspamd_stats: Output progress info on STDERR
- [Feature] Whitelist for emails module
- [Fix] Do not allow dependencies on self
- [Fix] Do not cache metric result
- [Fix] Do not trust all issuers as a client certificate
- [Fix] Fix dependencies in lua squeeze
- [Fix] Fix enabling/disabling squeezed rules
- [Fix] Fix enabling/disabling symbols
- [Fix] Fix external dependencies
- [Fix] Fix processing of a single compressed file
- [Fix] Fix some typos
- [Fix] Fix various modules in case of empty message
- [Fix] Handle callbacks that returns table of options
- [Fix] Improve cached action interaction
- [Fix] Make dynamic conf more NaN aware
- [Fix] Never hide actions from WebUI `configuration` tab
- [Project] Implementation of Lua rules squeezing
Postfix stable release 3.3.0 is available. This release ends support
for legacy release Postfix 2.11.
The main changes are:
* Dual license: in addition to the historical IBM Public License
1.0, Postfix is now also distributed with the more recent Eclipse
Public License 2.0. Recipients can choose to take the software
under the license of their choice. Those who are more comfortable
with the IPL can continue with that license.
* The postconf command now warns about unknown parameter names
in a Postfix database configuration file. As with other unknown
parameter names, these warnings can help to find typos early.
* Container support: Postfix 3.3 will run in the foreground with
"postfix start-fg". This requires that Postfix multi-instance
support is disabled (the default). To collect Postfix syslog
information on the container's host, mount the host's /dev/log
socket into the container, for example with "docker run -v
/dev/log:/dev/log ...other options...", and specify a distinct
Postfix syslog_name setting in the container (for example with
"postconf syslog_name=the-name-here").
* Milter support: applications can now send RET and ENVID parameters
in SMFIR_CHGFROM (change envelope sender) requests.
* Postfix-generated From: headers with 'full name' information
are now formatted as "From: name <address>" by default. Specify
"header_from_format = obsolete" to get the earlier form "From:
address (name)".
* Interoperability: when Postfix IPv6 and IPv4 support are both
enabled, the Postfix SMTP client will now relax MX preferences
and attempt to schedule similar numbers of IPv4 and IPv6
addresses. This works around mail delivery problems when a
destination announces lots of primary MX addresses on IPv6, but
is reachable only over IPv4 (or vice versa). The new behavior
is controlled with the smtp_balance_mx_inet_protocols parameter.
* Compatibility safety net: with compatibility_level < 1, the
Postfix SMTP server now warns for mail that would be blocked
by the Postfix 2.10 smtpd_relay_restrictions feature, without
blocking that mail. There still is a steady trickle of sites
that upgrade from an earlier Postfix version.
Action Mailer is a framework for designing email-service layers. These layers
are used to consolidate code for sending out forgotten passwords, welcome
wishes on signup, invoices for billing, and any other use case that requires
a written notification to either a person or another system.
Action Mailer is in essence a wrapper around Action Controller and the
Mail gem. It provides a way to make emails using templates in the same
way that Action Controller renders views using templates.
Additionally, an Action Mailer class can be used to process incoming email,
such as allowing a weblog to accept new posts from an email (which could even
have been sent from a phone).
This is for Ruby on Rails 5.1.
1.03 Thu Mar 15 21:55:30 2018
- update dovecot parser from dovecot version 2.3.0.1
- fix reading from uninitialized memory when formatting invalid address without user or host part
- fix formatting email address which user part starts with null byte
- do not generate invalid email addresses by format functions, rather return empty string
1.1.0:
+ Changed from distutils to setuptools because it's the future
+ Implement RFC 7601 SHOULD to ignore unknown method identifiers (2.7.6):
+ Discard unknown ptypes and associated properties
+ Added tests to document errors raised by different kinds of broken header
fields
Version 2.7.0 (2017-10-31)
Security:
* #1097 – SMTP security: prevent command injection via To/From addresses.
(jeremy)
Features:
* #647 – IMAP: specify IMAP server search charset with
Mail.find(search_charset: 'UTF-8'). (yalab)
* #650 - UTF-7 charset support. (johngrimes)
* #664 - RSpec: with_html and with_text matchers. (zakkie)
* #723 – IMAP: support `enable_starttls: true` for TLS upgrade on
non-IMAPS/SSL servers. (doits)
* #804 - Configurable SMTP open_timeout and read_timeout. (ankane)
* #853 - `Mail::Message#set_sort_order` overrides the default message part
sort order. (rafbm)
* #856 - Added :logger delivery method. (zacholauson)
* #900 - Support non-instance_eval builder API. Yield self to Mail.new if the
provided block takes any arguments. (taavo)
* #1065 - Require STARTTLS using :enable_starttls. (bk2204)
* #1002 - Transcoding replaces invalid chars with "�" instead of discarding
them. (kjg)
* #1053 - Ruby 2.4.0 compatibility. Fixnum+Bignum unified as
Integer. (peterkovacs)
* #1094 - Core extensions removal: Drop `String#at`, `from`, `last` and
`is_utf8?` since they are no longer used by Mail internals. (metcalf)
* #1095 - Core extensions removal: Drop `String#mb_chars`, `not_ascii_only?`,
`constantize`, `first`, `to` to avoid monkey patching the standard
library. (metcalf)
* #1111 - Mail::Field.parse API which deprecates calling Mail::Field.new with
unparsed header fields. (jeremy)
* #1117 - Configurable POP3 read_timeout. (hspazio)
Performance:
* #1059 - Switch from mime-types to mini_mime for a much smaller memory
footprint. (SamSaffron)
* #1119 - Speed up large attachment encoding by memoizing slow ASCII-only
checks. (dalibor)
Compatibility:
* #464 - Improve attachment filename detection by preferring
Content-Disposition filename. (lawrencepit)
* #535 - IMAP: fetch messages WITH IMAP FLAGS by passing a block with four
args. (lawrencepit)
* #558 - Parser: cope with unknown charsets in header fields by falling back
to ASCII. (boesemar)
* #655 - Sort attachments to the end of the parts list to work around email
clients that may mistake a text attachment for the message body. (npickens)
* #683 - SMTP: Work around Net::SMTP dot-stuffing bug with unterminated
newlines on Ruby 1.8 and 1.9. (yyyc514)
* #766 - No longer strip 'Subject: ' from legit subject lines. (grosser)
* #982 – Faithfully preserve unfolded whitespace rather than collapsing to a
single space. (jeremy)
* #1103 – Support parsing UTF-8 headers. Implements RFC 6532. (jeremy)
* #1106 – Limit message/rfc822 parts' transfer encoding per RFC 2046. (ahorek)
* #1112 – Support Windows-1258 charset by parsing it as Windows-1252 in
Ruby. (jeremy)
* #1114 – Setting `mail.body = …` on a multipart message now adds a new text
part instead of adding a raw MIME part. (jeremy)
* #1159 – Parse emails with n newlines so long as they have no binary
content. (jeremy)
Bugs:
* #539 - Fix that whitespace-only continued headers would be incorrectly
parsed as the break between headers and body. (ConradIrwin)
* #605 - Fix Mail::Address#name for nil addresses (peterkovacs)
* #684 - Fix recursively fetching attachments from an embedded message/rfc822
part whose Content-Type header has additional parameters. (vongruenigen)
* #689 - Fix Exim delivery method broken by #477 in 2.5.4. (jethrogb)
* #792 - Allow blank filenames in Content-Disposition field.
(robinroestenburg)
* #876 - Strip valid RFC-1342 separator characters between non-matching
encoded-words. (Caleb W. Corliss)
* #895 - Fix that Mail::Message#add_file was adding a stray filename
header. (kirikak2)
* #923 – Fix decoding nested quotes around non-US-ASCII addresses. (averell23)
* #978 - Fix for invalid chars being left in a string for invalid b_value from
encoding. (kjg)
* #996 - Fix that multipart/mixed emails with a delivery-status part could be
interpreted as bounces. (kjg)
* #998 - Fix header parameter parsing (such as attachment names) for values
encoded with a blank charset or language code. (kjg)
* #1000 - Fix header parameter parsing (such as attachment names) to transcode
to UTF-8 (kjg)
* #1003 - Fix decoding some b encoded headers on specific rubies that don't
account for lack of base64 padding (kjg)
* #1020 - Don't set SMTP verify mode to nil when config was not
provided. (jhass)
* #1023 - Fix double-quoting in display names. (garethrees)
* #1032 - Fix that comparing messages changed their raw Message-ID to their
parsed message_id. (bobjflong)
* #1074 - Fix that the first address in a list is dropped when a subsequent
address has non-US-ASCII characters. (domininik)
* #1107 - Fix Address#display_name and other formatting flip-flopping between
encoded and decoded forms depending on whether #encoded or #decoded was
called last. (jeremy)
* #1110 - Fix that Mail::Multibyte::Chars#initialize mutated its argument by
calling force_encoding on it. (jeremy)
* #1122 – Fix that tilde (~) shouldn't be escaped for Exim delivery. (Benabik)
* #1113 - Eliminate attachment corruption caused by CRLF conversion. (jeremy)
* #1131 - Fix that Message#without_attachments! didn't parse the remaining
parts. (jeremy)
* #1019 - Fix b value encoder incorrectly splitting multibyte characters.
(Kenneth-KT)
* #1157 - Fix base64 attachment transfer encoding being overridden by
quoted-printable. (dalibor)
- [Conf] Add bayes_expiry as explicit module
- [Conf] Adjust names and weights for neural network plugin
- [Conf] Change updates url
- [Conf] Default statistics is stored in Redis now
- [Conf] Disable fann_redis module by default
- [Conf] Fix default elastic configuration
- [Conf] Fix double quote position
- [Conf] Massive config rework for new structure of symbols and scores
- [Conf] Rename Rambler BLs as they are now Rspamd's ones
- [Conf] Use dedicated rspamd.com subdomains
- [Conf] Use more data from rspamd.com fuzzy storage
- [CritFix] Add sanity guards for badly broken HTML
- [CritFix] Another errors path handling fix
- [CritFix] Another portion of tokenization fixes
- [CritFix] Do not send reject messages after set reply
- [CritFix] Fix ARC chain verification
- [CritFix] Fix crash in milter errors handler
- [CritFix] Fix memory leak in spf caching logic
- [CritFix] Fix milter commands pipelining
- [CritFix] Fix newlines detection
- [CritFix] Fix semicolons parsing in the content type
- [CritFix] Plug memory leak in zstd protocol compression
- [Feature] Add ability to match score in force_actions module
- [Feature] Add aes-rng PRF to libottery
- [Feature] Add 'composites' debug module
- [Feature] Add concept of experimental modules
- [Feature] Add DKIM trace symbol
- [Feature] Add EBL to the default config
- [Feature] Add expected ip check for emails plugin
- [Feature] Add framework to manage Redis scripts
- [Feature] Add framing for the new reputation generic plugin
- [Feature] Add function to show plugins stat
- [Feature] Add gzip compression support for clickhouse module
- [Feature] Add gzip compression support for rspamd controller
- [Feature] Add gzip support when sending lua http requests
- [Feature] Add json output for rspamd_stats
- [Feature] Add method to do a synchronous Redis connection
- [Feature] Add method to get all content-type attributes in Lua
- [Feature] Add `-m` flag to configdump to show modules states
- [Feature] Add mime types to extensions map
- [Feature] Add more features to rescore utility
- [Feature] Add more gtube like patterns to test other spam actions
- [Feature] Add more metafunctions, improve logging
- [Feature] Add more text attributes
- [Feature] Add new configwizard command to rspamadm
- [Feature] Add new tooling for stats conversation
- [Feature] Add old groups migration tool
- [Feature] Add plugins state variable
- [Feature] Add preliminary ecdsa keys support in DKIM
- [Feature] Add preliminary support of idempotent symbols
- [Feature] Add Redis server wizard
- [Feature] Add routine to convert old style stats to a new one
- [Feature] Add some sanity checks for actions and controller
- [Feature] Add statistic convertation module to configwizard
- [Feature] Add sugestions logic to mempool allocator
- [Feature] Add support of config transform in Lua
- [Feature] Add timeout to rspamc when doing corpus test
- [Feature] Add tooling to convert bayes schemas
- [Feature] Add torch conditional to configuration
- [Feature] Add torch-decisiontree package
- [Feature] Add torch-optim contrib package
- [Feature] Add TTL autodetection
- [Feature] Add urls reputation to the reputation framework
- [Feature] Allow floating and negative values in expressions limits
- [Feature] Allow multiple CTs in full extensions map
- [Feature] Allow multiple fann rules
- [Feature] Allow randomly select User-Agent from a list
- [Feature] Allow rspamadm commands to export methods in Lua
- [Feature] Allow rule specific min_bytes in fuzzy check
- [Feature] Allow to adjust symbols scores from Lua
- [Feature] Allow to attach stat signature to messages
- [Feature] Allow to change SMTP from via milter headers
- [Feature] Allow to configure monitored
- [Feature] Allow to create directories in Lua API
- [Feature] Allow to disable torch and skip train samples for ANN
- [Feature] Allow to discard messages dynamically
- [Feature] Allow to enable/disable languages from the detector
- [Feature] Allow to generate DKIM keys from rspamadm API
- [Feature] Allow to get CPU flags from Lua
- [Feature] Allow to have high precision timestamps in logs
- [Feature] Allow to insert headers into specific position
- [Feature] Allow to limit redirector requests per task
- [Feature] Allow to load and use dynamic ANNs with torch
- [Feature] Allow to quarantine rejected messages using milter
interface
- [Feature] Allow to receive signing keys from mempool vars
- [Feature] Allow to reserve elements in libucl
- [Feature] Allow to reuse signal handlers chains
- [Feature] Allow to set custom mempool variables from settings
- [Feature] Allow to set headers from settings
- [Feature] Allow to set Settings-Id for all connections
- [Feature] Allow to skip real action and add a header instead
- [Feature] Allow to skip specific hashes in fuzzy storage
- [Feature] Allow to spawn asynchronous processes from Lua
- [Feature] Allow to specify number of threads for ANN learning
- [Feature] Allow to use global lua maps in settings
- [Feature] Allow to use postfilters in composites
- [Feature] Allow to verify signatures from HTTP headers in maps
- [Feature] Antivirus: ordered pattern matches
- [Feature] Authentication-Results: support hiding usernames
- [Feature] Automatically create tables in clickhouse
- [Feature] Catch next-to-last bad extension
- [Feature] Check cached maps more frequently
- [Feature] Check groups sanity
- [Feature] Deal with obscured URLs with @ symbols
- [Feature] Enhance task:store_in_file method
- [Feature] Export password encryption routines to Redis
- [Feature] Filter nan and inf when adding scores
- [Feature] Finalize 7zip files support
- [Feature] Further improvements in language detection
- [Feature] Further improvements in language detection algorithm
- [Feature] Generic key name expansion for Redis keys
- [Feature] Hash whitelist for fuzzy_check
- [Feature] Implement bayes signatures storage
- [Feature] Implement buckets for Redis backend
- [Feature] Implement DKIM reputation adjustments
- [Feature] Implement forked workers children monitoring
- [Feature] Implement headers flags in mime parser
- [Feature] Implement l1/l2 regularization against the current weights
- [Feature] Implement manual ANN train mode
- [Feature] Implement per-user ANN support
- [Feature] Implement torch based ANN learning
- [Feature] Implement upstreams logic for clickhouse exporter
- [Feature] Import torch to Rspamd...
- [Feature] Improve allocation policy when interacting with Lua
- [Feature] Improve Lua/C interaction in history_redis
- [Feature] Improve multiple fuzzy results combining
- [Feature] Improve parsing of DKIM keys: parse algorithm
- [Feature] Improve subprocesses termination handle
- [Feature] Improve symbol type parsing in Lua API
- [Feature] Metadata Exporter: e-Mail Alerts: support multiple
recipients; alerting senders/recipients/users
- [Feature] Milter headers: support adding/removing arbitrary headers
from config
- [Feature] More metatokens
- [Feature] Multimap: checking of symbol options
- [Feature] Multimap: template URL filter
- [Feature] New bayes expiry plugin
- [Feature] Periodically save rspamd stats to disk
- [Feature] Preliminary import of the elasticsearch module
- [Feature] Ratelimit: allow full addresses in whitelisted_rcpts
- [Feature] Ratelimit: support fetching limits from Redis
- [Feature] RBL: received: filtering by position & flags
- [Feature] Read global maps for lua
- [Feature] Redis settings: support checking multiple keys
- [Feature] Rework fann plugin to be a normal post-filter
- [Feature] Rework logging configuration for rspamadm case
- [Feature] Rework short hashes generation to avoid FP
- [Feature] Save real ucl types when exporting to Lua
- [Feature] Set TCP_NODELAY for milter sockets
- [Feature] Setup DKIM signing from configwizard
- [Feature] Skip certain symbols from ANN classify
- [Feature] Store plugins state
- [Feature] Support etag for HTTP maps
- [Feature] Support Expires header when using HTTP maps
- [Feature] Support sending given header multiple times in lua_http
- [Feature] Support sha512 in DKIM signatures
- [Feature] Try to detect HTML messages better
- [Feature] Use array instead of queue to reduce memory fragmentation
- [Feature] Use controller port by default when connecting to local IP
- [Feature] Use rdtsc where possible
- [Fix] Actively load skip hashes map in fuzzy storage
- [Fix] Add another workaround to display history properly
- [Fix] Add definition for old glib compatibility method
- [Fix] Add missing rspamadm control options to help
- [Fix] Add workaround for IPv6 in sendmail
- [Fix] Add workaround for system with non-XSI compatible tzset
- [Fix] Allow oversigning in DKIM signatures
- [Fix] Allow to check negative scores in force_actions
- [Fix] Allow to have negative actions limits
- [Fix] Allow to set any layers number for fann rules
- [Fix] Another fix for rdtcs
- [Fix] Another fix to lua xmlrpc
- [Fix] Another try to deal with #1998
- [Fix] Another try to fix#1998
- [Fix] Another try to fix threading in torch
- [Fix] Apply language detection when adding fuzzy hashes
- [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer
- [Fix] Authentication Results: Fix SPF smtp.mail_from
- [Fix] Auth-Results: Multiple DKIM signatures
- [Fix] Avoid changing content-transfer-encoding header's value
- [Fix] Better handling of the legacy protocol
- [Fix] Check decoded headers sanity (e.g. by excluding \0)
- [Fix] Check for magic when checking for an archive
- [Fix] Cleanup mess with groups
- [Fix] Clickhouse: Insertion in the symbols table
- [Fix] Crash in URL processing
- [Fix] Deal with another case when processing exceptions
- [Fix] Deal with deeply nested messages more aggressively
- [Fix] Deal with nan and inf encoding in json/ucl
- [Fix] Deal with non-key arguments in lua_redis.exec_script
- [Fix] Deal with unknown weight
- [Fix] Deal with URLs with no slashes after protocol
- [Fix] Deal with URLs wrapped in [] in text parts
- [Fix] Deal with zero scores symbols
- [Fix] Default monitoring domain for surbl plugin
- [Fix] Delay upstream re-resolving when one upstream is defined
- [Fix] Detection of maillist optimized and fixed
- [Fix] DKIM signing: allow for auth_only to be false
- [Fix] DMARC: require report_settings for sending reports only
- [Fix] Do not allow garbadge when checking url domain
- [Fix] Do not cache SPF records with PTR elements
- [Fix] Do not constantly re-resolve failed upstreams with a single
element
- [Fix] Do not crash if no words defined
- [Fix] Do not crash on empty subtype
- [Fix] Do not expose spamtrap messages to SMTP reply
- [Fix] Do not fail rbl plugin when there are no received or emails
- [Fix] Do not ignore short words
- [Fix] Do not include idempotent/nostat symbols to checksum
- [Fix] Do not override groups when converting metrics
- [Fix] Do not override unix socket group when group comes before
owner
- [Fix] Do not skip the last character
- [Fix] Do not spawn too many workers by default
- [Fix] Do not stop monitored on dns errors
- [Fix] Do not stop parsing headers on bad IP header
- [Fix] Do not strip last character in the last word
- [Fix] Do not treat script content as text
- [Fix] Do not try to connect to non-supported addresses
- [Fix] Do not try to dereference last character
- [Fix] Do not try to sign unknown domains
- [Fix] Don't use whitelist/greylist maps as regexp, but as map
- [Fix] Erase unknown HTML entities
- [Fix] Exim Received header protocol parsing
- [Fix] First load selector_map and path_map. And only return false
when domain not found if try_fallback is false
- [Fix] Fix a lot of FP in chartable in mixed languages
- [Fix] Fix ANN checks
- [Fix] Fix ANN loading logic
- [Fix] Fix another tokenization issue
- [Fix] Fix autolearn parameters reading
- [Fix] Fix bad archive characters stripping
- [Fix] Fix bad extension check
- [Fix] Fix bayes schema conversion
- [Fix] Fix blacklists and DMARC in whitelist
- [Fix] Fix brain-damaged torch build system
- [Fix] Fix build on FreeBSD
- [Fix] Fix clickhouse exporter
- [Fix] Fix clickhouse schema
- [Fix] Fix comparision
- [Fix] Fix composites processing
- [Fix] Fix connecting to a unix socket in rspamadm statconvert
- [Fix] Fix couple of warnings
- [Fix] Fix crashes in the rspamd_control path
- [Fix] Fix deletion from hash
- [Fix] Fix DKIM forgeries via multiple headers
- [Fix] FIx dynamic conf plugin
- [Fix] Fix emails detection
- [Fix] Fix empty headers simple canonicalization
- [Fix] Fix empty threshold check in greylisting module
- [Fix] Fix encrypted legacy reply in fuzzy storage
- [Fix] Fix enormous scores for R_WHITE_ON_WHITE
- [Fix] Fix exceptions list in surbl
- [Fix] Fix *_EXCESS_BASE64 rules
- [Fix] Fix expire rounding
- [Fix] Fix extra hits in PCRE mode for regular expressions
- [Fix] Fix format strings
- [Fix] Fix get_content method
- [Fix] Fix groups override when defining symbols
- [Fix] Fix learned count in new schema
- [Fix] Fix learn errors propagation
- [Fix] Fix loading of per-user redis backend for statistics
- [Fix] Fix logging buffer corruption in case of repeated messages
- [Fix] Fix lua cached elements invalidation
- [Fix] Fix merging of the implicit arrays
- [Fix] Fix mime_types scoring
- [Fix] Fix multiple headers in DKIM headers list
- [Fix] Fix null callee case in clang plugin
- [Fix] Fix obscured url in format user@@example.com
- [Fix] Fix parsing of the per-user script
- [Fix] Fix priorities in rspamd_update, disable rules execution
- [Fix] Fix processing of closed tags
- [Fix] Fix processing of idempotent rules when autolearn fails
- [Fix] Fix processing of multipart parts with no headers
- [Fix] Fix processing of skip-hashes in fuzzy storage
- [Fix] Fix PTR processing in SPF
- [Fix] Fix pushing country to clickhouse asn table
- [Fix] Fix random forests module
- [Fix] Fix real IP parsing for some strange Exim received
- [Fix] Fix Redis timeout setup
- [Fix] Fix reload crash when hyperscan is enabled
- [Fix] Fix reusing of redis connection after exec
- [Fix] Fix sanity checks on macro value
- [Fix] Fix setting of path and cpath for Lua
- [Fix] Fix setting of signals when spawning a thread
- [Fix] Fix text splitting: stack overflow (too many captures)
- [Fix] Fix ticks processing
- [Fix] Fix upstream addrs updating
- [Fix] Fix urls/emails distinguishing found in queries
- [Fix] Fix user settings check
- [Fix] Fix variable increment
- [Fix] Fix various issues in stat_convert
- [Fix] F-PROT Antivirus infection string for all known occurences
- [Fix] F-PROT Antivirus: only check return code to determine
infection
- [Fix] Further fixes around floating point expressions
- [Fix] Further fixes to ANN module
- [Fix] Further fixes to rescore tool
- [Fix] Further fixes to support ES 6
- [Fix] Further tokenization fixes
- [Fix] Greylisting set phase is not idempotent
- [Fix] Handle proxy copy errors
- [Fix] Header checks: Fix get_raw_header method
- [Fix] Header checks: REPLYTO_UNPARSEABLE rule
- [Fix] Kill spawned processes on termination
- [Fix] Load skip map from all processes as shared cache is
unavailable
- [Fix] Lowercase HTTP headers to make them searchable from Lua
- [Fix] Lowercase words
- [Fix] Lua_http: freeing
- [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid
"rspamd_config_read: rcl parse error: cannot init lua file [...]
module 'lpeg' not found"
- [Fix] Map absence is not an error
- [Fix] Metadata exporter: check IP sanity
- [Fix] Milter headers: custom headers: removing headers
- [Fix] Milter headers: skip_local / skip_authenticated settings
- [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first
- [Fix] mime_types: fix next-to-last extension length check
- [Fix] More hacks to deal with old configs
- [Fix] Move composites second pass to the dedicated stage
- [Fix] Multimap: received: filtering of artificial header
- [Fix] Multiple fixes in torch based ANN plugins
- [Fix] Once more fix bad extension check
- [Fix] Optimize rspamd_fstring_t reallocations
- [Fix] options.local_networks setting
- [Fix] Parse HREF urls without explicit prefix
- [Fix] Plan new event on HTTP errors
- [Fix] Plug another possible memory leak
- [Fix] Plug memory leak
- [Fix] Plug memory leak in lua_tcp
- [Fix] Plug memory leak when setting email addresses from Lua
- [Fix] Propagate learn/stat errors more precisely
- [Fix] Ratelimit: fix whitelisted_rcpts matching
- [Fix] Ratelimit: lowercase email addresses
- [Fix] RBL: received: deal with missing data
- [Fix] Rebalance and slightly rework MX check plugin
- [Fix] Redis key expansion: EVAL: deal with strings
- [Fix] Redis script loading in DMARC; URL tags; URL reputation
- [Fix] Reject invalid bh for DKIM signatures earlier
- [Fix] Relax pem signature detection
- [Fix] Relax unicode properties requirements for chartable module
- [Fix] Remove extra noise from dkim and arc signing
- [Fix] Remove hop-by-hop headers in proxy
- [Fix] Remove incorrect method `task:set_metric_subject`
- [Fix] Replace space like characters in headers with plain space
- [Fix] Restore old style ratelimits support
- [Fix] Rework elasticsearch plugin
- [Fix] Rewriting subjects via force actions module
- [Fix] RPM postinstall
- [Fix] Sanitize IP in history redis
- [Fix] Select the correct signature when doing simple canon
- [Fix] Set CLOEXEC flag on files opened
- [Fix] Setting check_local / check_authed in plugins
- [Fix] Settings: avoid checking invalid IP
- [Fix] Settings: header: deal with multiple settings
- [Fix] Skip checks if both extensions are not bad
- [Fix] Skip nostat tokens when get number of tokens
- [Fix] Some more fixes towards emails detection
- [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't
match
- [Fix] Stop using of g_slice...
- [Fix] Switch rspamadm logging to message level
- [Fix] Symbol 'FANNR_SPAM' has its score defined..
- [Fix] Table parameter for rspamd_config:add_doc()
- [Fix] Treat 'rewrite subject' as spam action
- [Fix] Try harder in passing IPv6 addresses
- [Fix] Try harder to find rfc822 notifications
- [Fix] Try harder to find urls
- [Fix] Use decoded values when parsing mime addresses
- [Fix] Use full URL when making an HTTP request
- [Fix] Use greylisting threshold in greylisting module
- [Fix] Use n_words attribute from ngramms
- [Fix] Use raw urls when sending requests to redirector
- [Fix] Use the right boolean operator on error check
- [Fix] Use weight from map for fuzzy scoring
- [Fix] Various fixes to elastic plugin
- [Fix] Various fixes to fann_redis instantiation
- [Fix] Various improvements in language detection
- [Fix] Virus infection string for F-PROT Antivirus
- [Fix] Virus infetction string for F-PROT Antivirus
- [Fix] WebUI: use relative path for savemap
- [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the
rule
- [Fix] Write configuration changes as UCL config
- [Project] Add detection logic for words
- [Project] Add fast debug logging infrastructure
- [Project] Add more flags to languages
- [Project] Add n-gramms data files
- [Project] Add ngramms frequencies detector
- [Project] Add random words selection logic
- [Project] Add unigramms to language detection as well
- [Project] Convert all C modules to fast debug infrastructure
- [Project] Detect some languages based on unicode script
- [Project] Enable fast debug lookup for some modules
- [Project] Enable language detector init in scanner workers
- [Project] Further improvements to language detector
- [Project] Implement logic of ngramms application
- [Project] Improve weighting in lang_detection
- [Project] Initialize language detector
- [Project] Preliminary version of ngramms based language detector
- [Project] Preliminary version of the new stat_convert
- [Project] Remove old language detector
- [Project] Rework language detection ngramms structure
- [Project] Start language detection project
- [Project] Start rework of language detection to improve quality
- [Project] Use fast debug logging check
- [Rework] Add frame for new reputation based IP score module
- [Rework] Continue stat_convert rework task
- [Rework] Implement new version of fuzzy replies
- [Rework] Improve readability of xmlrpc API
- [Rework] Kill metrics!11
- [Rework] Ratelimit module
- [Rework] Rename fann_redis to neural plugin
- [Rework] Reorganize mime_types module
- [Rework] Rework rescore utility
- [Rework] Rewrite model and learning logic for rescore
- [Rework] Run post-loads when all initialization is completed
- [Rework] Simplify lua path initialization
- [Rework] Start major stat_convert rework
- [Rework] Start mempool fragmentation reduce project
- [Rework] Start moving of fann redis to torch
- [Rework] Stop embedding rspamadm scripts into C
- [Rework] Use floating point arithmetics in Rspamd expressions
- [Rework] Use frequencies distribution in language detector
- [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages
- [WebUI] Compact graph selectors
- [WebUI] Escape strings inside HTML in history
- [WebUI] Fix message count in throughput summary
- [WebUI] Fix NaNs display on Throughput graph
- [WebUI] Migrate widgets to D3 v4
- [WebUI] Restore passwordless login support
- [WebUI] Show symbol descriptions as tooltips in history
- [WebUI] Stop using commas in pie chart tooltips
- [WebUI] Update D3 and jQuery
- [WebUI] Update D3Evolution 1.0.0 -> 1.1.0
pkgsrc changes:
- Update patch-ca to avoid patching unused by pkgsrc `uninstall-*'
targets (not needed) and adjust `installdirs' target to create
`egdir' (`share/examples/nmh')
Changes:
1.7.1
-----
1.7.1 is a patch release for 1.7, and includes fixes to a number of
significant bugs we have discovered since releasing 1.7. Specifically,
this release includes the following bug fixes:
- A significant memory leak in scan(1)
- rcvdist(1) not passing arguments to post(8) correctly
- Number formatting functions in the format engine were not truncating
numbers correctly
- Various fixes to the test suite
Exim version 4.90.1
JH/03 Fix pgsql lookup for multiple result-tuples with a single column.
Previously only the last row was returned.
JH/04 Bug 2217: Tighten up the parsing of DKIM signature headers. Previously
we assumed that tags in the header were well-formed, and parsed the
element content after inspecting only the first char of the tag.
Assumptions at that stage could crash the receive process on malformed
input.
JH/05 Bug 2215: Fix crash associated with dnsdb lookup done from DKIM ACL.
While running the DKIM ACL we operate on the Permanent memory pool so that
variables created with "set" persist to the DATA ACL. Also (at any time)
DNS lookups that fail create cache records using the Permanent pool. But
expansions release any allocations made on the current pool - so a dnsdb
lookup expansion done in the DKIM ACL releases the memory used for the
DNS negative-cache, and bad things result. Solution is to switch to the
Main pool for expansions.
While we're in that code, add checks on the DNS cache during store_reset,
active in the testsuite.
Problem spotted, and debugging aided, by Wolfgang Breyha.
JH/06 Fix issue with continued-connections when the DNS shifts unreliably.
When none of the hosts presented to a transport match an already-open
connection, close it and proceed with the list. Previously we would
queue the message. Spotted by Lena with Yahoo, probably involving
round-robin DNS.
JH/07 Bug 2214: Fix SMTP responses resulting from non-accept result of MIME ACL.
Previously a spurious "250 OK id=" response was appended to the proper
failure response.
JH/10 Bug 2223: Fix mysql lookup returns for the no-data case (when the number of
rows affected is given instead).
JH/12 Bug 2230: Fix cutthrough routing for nonfirst messages in an initiating
SMTP connection. Previously, when one had more receipients than the
first, an abortive onward connection was made. Move to full support for
multiple onward connections in sequence, handling cutthrough connection
for all multi-message initiating connections.
JH/13 Bug 2229: Fix cutthrough routing for nonstandard port numbers defined by
routers. Previously, a multi-recipient message would fail to match the
onward-connection opened for the first recipient, and cause its closure.
JH/14 Bug 2174: A timeout on connect for a callout was also erroneously seen as
a timeout on read on a GnuTLS initiating connection, resulting in the
initiating connection being dropped. This mattered most when the callout
was marked defer_ok. Fix to keep the two timeout-detection methods
separate.
HS/01 Fix Buffer overflow in base64d() (CVE-2018-6789)
JH/16 Fix bug in DKIM verify: a buffer overflow could corrupt the malloc
metadata, resulting in a crash in free().
PP/01 Fix broken Heimdal GSSAPI authenticator integration.
Broken in f2ed27cf5, missing an equals sign for specified-initialisers.
Broken also in d185889f4, with init system revamp.
Changelog:
Fixed Searching message bodies of messages in local folders, including
filter and quick filter operations, not working reliably:
Content not found in base64-encode message parts, non-ASCII text
not found and false positives found.
Fixed Defective messages (without at least one expected header) not shown
in IMAP folders but shown on mobile devices
Fixed Calendar: Unintended task deletion if numlock is enabled
Fixed Various security fixes
Security fixes:
#CVE-2018-5095: Integer overflow in Skia library during edge builder allocation
#CVE-2018-5096: Use-after-free while editing form elements
#CVE-2018-5097: Use-after-free when source document is manipulated during XSLT
#CVE-2018-5098: Use-after-free while manipulating form input elements
#CVE-2018-5099: Use-after-free with widget listener
#CVE-2018-5102: Use-after-free in HTML media elements
#CVE-2018-5103: Use-after-free during mouse event handling
#CVE-2018-5104: Use-after-free during font face manipulation
#CVE-2018-5117: URL spoofing with right-to-left text aligned left-to-right
#CVE-2018-5089: Memory safety bugs fixed in Firefox 58, Firefox ESR 52.6,
and Thunderbird 52.6
Small patch release to fix the worst bugs in v2.3.0. v2.3.1 is coming in about a month with a lot more changes.
* CVE-2017-15130: TLS SNI config lookups may lead to excessive
memory usage, causing imap-login/pop3-login VSZ limit to be reached
and the process restarted. This happens only if Dovecot config has
local_name { } or local { } configuration blocks and attacker uses
randomly generated SNI servernames.
* CVE-2017-14461: Parsing invalid email addresses may cause a crash or
leak memory contents to attacker. For example, these memory contents
might contain parts of an email from another user if the same imap
process is reused for multiple users. First discovered by Aleksandar
Nikolic of Cisco Talos. Independently also discovered by "flxflndy"
via HackerOne.
* CVE-2017-15132: Aborted SASL authentication leaks memory in login
process.
* Linux: Core dumping is no longer enabled by default via
PR_SET_DUMPABLE, because this may allow attackers to bypass
chroot/group restrictions. Found by cPanel Security Team. Nowadays
core dumps can be safely enabled by using "sysctl -w
fs.suid_dumpable=2". If the old behaviour is wanted, it can still be
enabled by setting:
import_environment=$import_environment PR_SET_DUMPABLE=1
- imap-login with SSL/TLS connections may end up in infinite loop
1.02 Sat Feb 03 13:41:38 2018
- add support for parsing and generating addresses with nul character
- fix function compose_address when both user and host contains non-ASCII 8bit characters
- fix possible memory leak in dovecot parser
Update mail/postfix to 3.2.5.
[An on-line version of this announcement will be available at
http://www.postfix.org/announcements/postfix-3.2.4.html]
This announcement concerns fixes for problems that were introduced
with Postfix 3.0 and later. Older supported releases are unaffected.
Fixed in Postfix 3.1 and later:
* DANE interoperability. Postfix builds with OpenSSL 1.0.0 or
1.0.1 failed to send email to some sites with "TLSA 2 X X" DNS
records associated with an intermediate CA certificate. Problem
report and initial fix by Erwan Legrand.
Fixed in Postfix 3.0 and later:
* Missing dynamicmaps support in the Postfix sendmail command.
This broke authorized_submit_users settings that use a
dynamically-loaded map type. Problem reported by Ulrich Zehl.
2018-02-23 Richard Russon <rich@flatcap.org>
* Features
- browser: `<goto-parent>` function bound to "p"
- editor: `<history-search>` function bound to "Ctrl-r"
- Cygwin support: https://www.neomutt.org/distro/cygwin
- OpenSUSE support: https://www.neomutt.org/distro/suse
- Upstream Homebrew support: Very soon - https://www.neomutt.org/distro/homebrew
* Bug Fixes
- gmail server-size search
- nested-if: correctly handle "<" and ">" with %?
- display of special chars
- lua: enable myvars
- for pgpewrap in default gpg.rc
- reply_regexp which wasn't formatted correctly.
- parsing of urls containing '?'
- out-of-bounds read in mutt_str_lws_len
* Translations
- Review fuzzy lt translations
- Updated French translation
* Website
- Installation guide for Cygwin
- Installation guide for OpenSUSE
- Installation guide for CRUX
* Build
- check that DTDs are installed
- autosetup improvements
- option for which version of bdb to use
- drop test for resizeterm -- it's always present
* Code
- split if's containing assignments
- doxygen: add/improve comments
- rename functions / parameters for consistency
- add missing {}s for clarity
- move functions to library
- reduce scope of variables
- boolify more variables
- iwyu: remove unnecessary headers
- name unicode chars
- tailq: migrate parameter api
- md5: refactor and tidy
- rfc2047: refactor and tidy
- buffer: improvements
- create unit test framework
- fix several coverity defects
* Upstream
- Fix s/mime certificate deletion bug
- Disable message security if the backend is not available
- Fix improper signed int conversion of IMAP uid and msn values
- Change imap literal counts to parse and store unsigned ints
- Fix imap status count range check
- cmd_handle_fatal: make error message a bit more descriptive
- Create pgp and s/mime default and sign_as key vars
- Add missing setup calls when resuming encrypted drafts
- mutt_pretty_size: show real number for small files
- examine_directory: set directory/symlink size to zero
- Add history-search function, bound to ctrl-r
- Avoid a potential integer overflow if a Content-Length value is huge
- Fix build issue with redefining the "accept" function.
- Added support for whitelists in the rbl plugin.
- Added option to skip the Received header for authenticated connections.
2.0.2 (2017-12-14)
* Fix treatment of No_Mail configuration parameter so that specifying
No_Mail = False (the default) does not cause incorrect results
* Conditionally import authres is Header_Type is AR and raise an error if it
is missing (sorry pep-8) to avoid cases where users change the config
and suddenly it doesn't work for an example, see:
https://bugzilla.redhat.com/show_bug.cgi?id=1208876
* Update and correct Mail_From_pass_restriction description in
policyd-spf.conf(5 ()
* Update HELO checking default option in policyd-spf.conf(5)
* Note that SPF_Not_Pass is not consistent with RFC 7208 in the HELO
checking section of policyd-spf.conf(5) - already documented for Mail From
1.0.2:
+ Added DKIM 'a' property so signature algorithm can be reported as proposed
for inclusion in draft-ietf-dmarc-rfc7601bis (experimental)
+ Added match_signature_algorithm to the DKIMAuthenticationResult class to
make it easier to find the correct DKIM result based on both domain and
algorithm
+ Added DKIM 's' property so signature algorithm can be reported as proposed
for inclusion in draft-ietf-dmarc-rfc7601bis (experimental)
- [CritFix] Add sanity guards for badly broken HTML
- [CritFix] Another errors path handling fix
- [CritFix] Fix ARC chain verification
- [CritFix] Fix crash in milter errors handler
- [Feature] Allow to insert headers into specific position
- [Feature] Allow to receive signing keys from mempool vars
- [Feature] Authentication-Results: support hiding usernames
- [Fix] Another try to deal with #1998
- [Fix] Another try to fix#1998
- [Fix] Better handling of the legacy protocol
- [Fix] Check decoded headers sanity (e.g. by excluding \0)
- [Fix] Deal with nan and inf encoding in json/ucl
- [Fix] Deal with URLs wrapped in [] in text parts
- [Fix] DKIM signing: allow for auth_only to be false
- [Fix] Do not crash on empty subtype
- [Fix] Do not fail rbl plugin when there are no received or emails
- [Fix] Do not skip the last character
- [Fix] Do not try to dereference last character
- [Fix] Do not try to sign unknown domains
- [Fix] Exim Received header protocol parsing
- [Fix] First load selector_map and path_map. And only return false
when domain not found if try_fallback is false
- [Fix] Fix bad archive characters stripping
- [Fix] Fix comparision
- [Fix] Fix connecting to a unix socket in rspamadm statconvert
- [Fix] Fix empty headers simple canonicalization
- [Fix] Fix extra hits in PCRE mode for regular expressions
- [Fix] Fix parsing of the per-user script
- [Fix] Fix processing of skip-hashes in fuzzy storage
- [Fix] Fix Redis timeout setup
- [Fix] Fix sanity checks on macro value
- [Fix] Fix text splitting: stack overflow (too many captures)
- [Fix] Fix urls/emails distinguishing found in queries
- [Fix] F-PROT Antivirus: only check return code to determine
infection
- [Fix] Metadata exporter: check IP sanity
- [Fix] Multimap: received: filtering of artificial header
- [Fix] Plan new event on HTTP errors
- [Fix] Plug another possible memory leak
- [Fix] Remove hop-by-hop headers in proxy
- [Fix] Sanitize IP in history redis
- [Fix] Setting check_local / check_authed in plugins (#1954)
- [Fix] Settings: avoid checking invalid IP (#1981)
- [Fix] Try harder in passing IPv6 addresses
- [Fix] WebUI: use relative path for savemap (#1943)
- [WebUI] Fix message count in throughput summary (#1724)
- [WebUI] Fix NaNs display on Throughput graph
- [WebUI] Restore passwordless login support (#2003)
use same PKG_OPTIONS_VAR as imap-uw to determine whether the build
needs to include kerberos support; this makes this extension actually
build against such imap-uw
bump PKGREVISION
when EXTRAAUTHENTICATORS is passed as MAKE_FLAGS, it ends up being
doubled, mkauths then generates auths.c with doubled auth_gss.c and
auth_mit.c twice, triggering duplicate definition errors with clang
9.0.0; pass via MAKE_ENV instead
bump PKGREVISION
Upstream changes:
version 2.20: Mon 22 Jan 18:14:44 CET 2018
Improvements:
- rewrite doc syntax to my current standard style.
- text corrections rt.cpan.org#123823 [Ville Skytt瓣]
- text corrections rt.cpan.org#123824 [Ville Skytt瓣]
- convert to GIT
- move to GitHUB
1.6.5: 22 Oct 2017
- [CritFix] Another portion of tokenization fixes
- [CritFix] Fix memory leak in spf caching logic
- [CritFix] Fix milter commands pipelining
- [CritFix] Fix newlines detection
- [Feature] Filter nan and inf when adding scores
- [Feature] Implement headers flags in mime parser
- [Feature] Support Expires header when using HTTP maps
- [Fix] Actively load skip hashes map in fuzzy storage
- [Fix] Add workaround for IPv6 in sendmail
- [Fix] Authentication Results: Fix SPF smtp.mail_from
- [Fix] Check for magic when checking for an archive
- [Fix] Deal with another case when processing exceptions
- [Fix] Deal with URLs with no slashes after protocol
- [Fix] Do not allow garbadge when checking url domain
- [Fix] Do not ignore short words
- [Fix] Do not strip last character in the last word
- [Fix] Do not treat script content as text
- [Fix] Erase unknown HTML entities
- [Fix] Fix another tokenization issue
- [Fix] Fix DKIM forgeries via multiple headers
- [Fix] Fix emails detection
- [Fix] Fix empty threshold check in greylisting module
- [Fix] Fix enormous scores for R_WHITE_ON_WHITE
- [Fix] Fix loading of per-user redis backend for statistics
- [Fix] Fix multiple headers in DKIM headers list
- [Fix] Fix obscured url in format user@@example.com
- [Fix] Further tokenization fixes
- [Fix] Load skip map from all processes as shared cache is
unavailable
- [Fix] Lowercase words
- [Fix] Milter headers: skip_local / skip_authenticated settings
- [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first
- [Fix] Ratelimit: fix whitelisted_rcpts matching
- [Fix] Some more fixes towards emails detection
- [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't
match
- [Fix] Use greylisting threshold in greylisting module
1.6.4: 10 Sep 2017
- [Feature] Add method to get all content-type attributes in Lua
- [Feature] Add some sanity checks for actions and controller
- [Feature] Allow randomly select User-Agent from a list
- [Feature] Deal with obscured URLs with @ symbols
- [Feature] Milter headers: support adding/removing arbitrary headers
from config
- [Fix] Add another workaround to display history properly
- [Fix] Add missing rspamadm control options to help
- [Fix] Auth-Results: Multiple DKIM signatures
- [Fix] Crash in URL processing
- [Fix] Default monitoring domain for surbl plugin
- [Fix] Detection of maillist optimized and fixed
- [Fix] Do not cache SPF records with PTR elements
- [Fix] Fix blacklists and DMARC in whitelist
- [Fix] Fix exceptions list in surbl
- [Fix] Fix processing of closed tags
- [Fix] Fix PTR processing in SPF
- [Fix] Lowercase HTTP headers to make them searchable from Lua
- [Fix] options.local_networks setting
- [Fix] Ratelimit: lowercase email addresses
- [Fix] Rebalance and slightly rework MX check plugin
- [Fix] Redis script loading in DMARC; URL tags; URL reputation
- [Fix] Reject invalid bh for DKIM signatures earlier
- [Fix] Remove incorrect method `task:set_metric_subject`
- [Fix] Rewriting subjects via force actions module
- [Fix] RPM postinstall
- [Fix] Treat 'rewrite subject' as spam action
- [Fix] Try harder to find urls
- [Fix] Use full URL when making an HTTP request
- [Fix] Use raw urls when sending requests to redirector
- [Fix] Use weight from map for fuzzy scoring
- [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages
1.6.3: 26 Jul 2017
- [CritFix] Fix semicolons parsing in the content type
- [Feature] Add EBL to the default config
- [Feature] Allow to configure monitored
- [Feature] Allow to skip specific hashes in fuzzy storage
- [Feature] Multimap: checking of symbol options
- [Feature] Redis settings: support checking multiple keys
- [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer
- [Fix] Avoid changing content-transfer-encoding header's value
- [Fix] Don't use whitelist/greylist maps as regexp, but as map
- [Fix] Fix get_content method
- [Fix] Header checks: Fix get_raw_header method
- [Fix] Header checks: REPLYTO_UNPARSEABLE rule
- [Fix] Lua_http: freeing
- [Fix] Milter headers: custom headers: removing headers
- [Fix] Parse HREF urls without explicit prefix
- [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the
rule
- [WebUI] Escape strings inside HTML in history
1.6.2: 08 Jul 2017
- [Conf] Remove Rambler email bl for now
- [Conf] Switch RAMBLER_URIBL to a locally managed source
- [CritFix] Switch from ragel to C for Content-Type parsing
- [Feature] Add `-e` option for lua_repl
- [Feature] Add per-domain emails normalisation rules
- [Feature] Add sessions cache to debug dangling sessions
- [Feature] Add short_text_direct_hash for fuzzy check module
- [Feature] Add text_part:get_stats function
- [Feature] Allow to add custom processing script for surbl
- [Feature] Allow to check reply-to email
- [Feature] Allow to customize spam header, remove existing spam
headers
- [Feature] Allow to disable specific workers in the config
- [Feature] Allow to discard messages instead of rejection
- [Feature] Allow to specify custom delimiter in emails plugin
- [Feature] Allow to specify custom User-Agent for rspamc
- [Feature] Allow to store symbols data in Clickhouse
- [Feature] Allow to use HTTPS when connecting to Clickhouse
- [Feature] Enable sessions cache tracking for milter connections
- [Feature] Implement per-line mode in lua_repl (like `perl -p`)
- [Feature] Implement rdns-curve plugin based on rspamd cryptobox
- [Feature] Improve maps cached data lifetime
- [Feature] Improve maps checking frequency
- [Feature] Improve monitored timeouts logic
- [Feature] milter_headers: add `extended_headers_rcpt` option
- [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility
headers
- [Feature] Milter headers: remove-header routine
- [Feature] Multimap: received filters for extracting TLDs from
hostnames
- [Feature] Normalize email aliases in emails module
- [Feature] Re-add rambler email bl (as hashed list)
- [Feature] Reload file maps more frequently
- [Feature] Rework newlines strip parser one more time
- [Feature] Skip updates for messages scanned via controller
- [Feature] Split long DKIM public keys
- [Feature] Store more data when stripping newlines
- [Feature] Support SPF macros transformations
- [Feature] Support suppressing DMARC reports for some domains
- [Fix] Add missing `break` statement
- [Fix] Allow modifiers in SPF macros
- [Fix] DKIM sign tools: edge-cases around use_esld
- [Fix] Do not cache SPF records with macros
- [Fix] Do not overwrite score when setting pre-action
- [Fix] Fix comparison logic
- [Fix] Fix DKIM base64 folding for milter flagged messages
- [Fix] Fix emails module configuration
- [Fix] Fix folding for arc headers when milter interface is used
- [Fix] Fix gmail dots removal
- [Fix] Fix rspamc detection in greylist module
- [Fix] Fix some more issues with HTTP maps
- [Fix] Milter sessions can live forever
- [Fix] Normalize fuzzy probability better
- [Fix] Plug memory leak
- [Fix] RBL: Fixed hashed email address lookups
- [Fix] Try to deal with brain-damaged milter behaviour
- [Fix] Use `\n` to fold headers for milter
- [Rework] Allow to use custom callback for monitored checks
- [Rework] Further steps towards one process monitoring
- [Rework] Send health checks from a single worker
- [WebUI] Round-up throughput summary values
Notmuch 0.26 (2018-01-09)
=========================
Command Line Interface
----------------------
Support for re-indexing existing messages
There is a new subcommand, `notmuch reindex`, which re-indexes all
messages matching supplied search terms. This permits users to
change the way specific messages are indexed.
Note that for messages with multiple variants in the message
archive, the recorded Subject: of may change upon reindexing,
depending on the order in which the variants are indexed.
Improved error reporting in notmuch new
Give more details when reporting certain Xapian exceptions.
Support maildir synced tags in `new.tags`
Tags `draft`, `flagged`, `passed`, and `replied` are now supported
in `new.tags`. The tag `unread` is still special in the presence of
maildir syncing, and will be added for files in `new/` regardless of
the setting of `new.tags`.
Support /regex/ in new.ignore
Files and directories may be ignored based on regular expressions.
Allow `notmuch insert --folder=""`
This inserts into the top level folder.
Strip trailing '/' from folder path for notmuch insert
This prevents a potential problem with duplicated database records.
New option --output=address for notmuch address
Make `notmuch show` more robust against deleting duplicate files
The option --decrypt now takes an explicit argument
The --decrypt option to `notmuch show` and `notmuch reply` now takes
an explicit argument. If you were used to invoking `notmuch show
--decrypt`, you should switch to `notmuch show --decrypt=true`.
Boolean and keyword arguments now take a `--no-` prefix
Encrypted Mail
--------------
Indexing cleartext of encrypted e-mails
It's now possible to include the cleartext of encrypted e-mails in
the notmuch index. This makes it possible to search your encrypted
e-mails with the same ease as searching cleartext. This can be done
on a per-message basis by passing --decrypt=true to indexing
commands (new, insert, reindex), or by default by running "notmuch
config set index.decrypt true".
Encrypted messages whose cleartext is indexed will typically also
have their session keys stashed as properties associated with the
message. Stashed session keys permit rapid rendering of long
encrypted threads, and disposal of expired encryption-capable keys.
If for some reason you want cleartext indexing without stashed
session keys, use --decrypt=nostash for your indexing commands (or
run "notmuch config set index.decrypt nostash"). See `index.decrypt`
in notmuch-config(1) for more details.
Note that stashed session keys permit reconstruction of the
cleartext of the encrypted message itself, and the contents of the
index are roughly equivalent to the cleartext as well. DO NOT USE
this feature without considering the security of your index.
Emacs
-----
Guard against concurrent searches in notmuch-tree
Use make-process when available
This allows newer Emacs to separate stdout and stderr from the
notmuch command without using temporary files.
Library Changes
---------------
Indexing files with duplicate message-id
Files with duplicate message-id's are now indexed, and searchable
via terms and phrases. There are known issues related to
presentation of results and regular-expression search, but in
principle no mail file should be completely unsearchable now.
New functions to count files
Two new functions in the libnotmuch API:
`notmuch_message_count_files`, and `notmuch_thread_get_total_files`.
New function to remove properties
A new function was added to the libnotmuch API to make it easier to
drop all properties with a common pattern:
`notmuch_message_remove_all_properties_with_prefix`
Change of return value of `notmuch_thread_get_authors`
In certain corner cases, `notmuch_thread_get_authors` previously
returned NULL. This has been replaced by an empty string, since the
possibility of NULL was not documented.
Transition `notmuch_database_add_message` to `notmuch_database_index_file`
When indexing an e-mail message, the new
`notmuch_database_index_file` function is the preferred form, and
the old `notmuch_database_add_message` is deprecated. The new form
allows passing a set of options to the indexing engine, which the
operator may decide to change from message to message.
Test Suite
----------
Out-of-tree builds
The test suite now works properly with out-of-tree builds, i.e. with
separate source and build directories. The --root option to tests
has been dropped. The same can now be achieved more reliably using
out-of-tree builds.
Python Bindings
---------------
Python bindings specific Debian packaging is removed
The bindings have been build by the top level Debian packaging for a
long time, and `bindings/python/debian` has bit-rotted.
Open mail files in binary mode when using Python 3
This avoids certain encoding related crashes under Python 3.
Add python bindings for `notmuch_database_{get,set}_config*`
Optional `decrypt_policy` flag is available for notmuch.database().index_file()
nmbug
-----
nmbug's internal version increases to 0.3 in this notmuch release.
User-facing changes with this notmuch release:
* Accept failures to unset `core.worktree` in `clone`, which allows
nmbug to be used with Git 2.11.0 and later.
* Auto-checkout in `clone` if it wouldn't clobber existing content,
which makes the initial clone more convenient.
* Only error for invalid diff lines in `tags/`, which allows for
`README`s and similar in nmbug repositories.
Documentation
-------------
New man page: notmuch-properties(7)
This new page to the manual describes common conventions for how
properties are used by libnotmuch, the CLI, and associated programs.
External projects that use properties are encouraged to claim their
properties and conventions here to avoid collisions.
Upstream changes:
version 2.17: Fri Jan 26 23:42:01 CET 2018
Fixes:
- when picking a preferred type for an extension, do prefer the type
with the same minor-name. Issue triggered by [Henry van Styn]
- remove iana obsoleted types
version 2.16: Tue 23 Jan 12:14:39 CET 2018
Fixes:
- collecting of IANA info has stalled: logic rewritten
Discovered by [Julien Lüthi]
Improvements:
- move scripts and source files into MANIFEST.extra
- update types and extensions
version 2.15: Fri 19 Jan 17:23:56 CET 2018
Improvements:
- moved to GIT and GitHUB.
Changelog:
Fix
This releases fixes the "Mailsploit" vulnerability and other vulnerabilities
detected by the "Cure53" audit. For details and various other security
fixes see here.
CVE-2017-7845: Buffer overflow when drawing and validating elements with
ANGLE library using Direct 3D 9
CVE-2017-7846: JavaScript Execution via RSS in mailbox:// origin
CVE-2017-7847: Local path string can be leaked from RSS feed
CVE-2017-7848: RSS Feed vulnerable to new line Injection
CVE-2017-7829: Mailsploit part 1: From address with encoded null character
is cut off in message header display
This library validates that address are of the form x@y.com. This is the sort
of validation you would want for a login form on a website.
Key features:
* Good for validating email addresses used for logins/identity.
* Friendly error messages when validation fails (appropriate to show to end
users).
* (optionally) Checks deliverability: Does the domain name resolve?
* Supports internationalized domain names and (optionally) internationalized
local parts.
* Normalizes email addresses (super important for internationalized addresses!).
Version 0.52
* Internet connection tests were declared in the wrong order
Version 0.51
* Fix for older versions of perl
* Tests no longer fail with no internet connection
Notmuch 0.25.3 (2017-12-08)
===========================
Emacs
-----
Extend mitigation (disabling handling x-display in text/enriched) for
Emacs bug #28350 to Emacs versions before 24.4 (i.e. without
`advice-add`).
Command Line Interface
----------------------
Correctly report userid validity. Fix test suite failure for GMime >=
3.0.3. This change raises the minimum supported version of GMime 3.x
to 3.0.3.
- feature request: added record_mailbox configuration parameter, to
allow turning off the header getmail adds with this information.
Thanks: Daniel Kahn Gillmor, Osamu Aoki, Josh Triplett.
Changelog v0.5.0.1:
- imap4flags extension: Fix binary corruption occurring when
setflag/addflag/removeflag flag-list is a variable.
- sieve-extprograms plugin: Fix segfault occurring when used in
IMAPSieve context.
The actual fix as been done by "pkglint -F */*/buildlink3.mk", and was
reviewed manually.
There are some .include lines that still are indented with zero spaces
although the surrounding .if is indented. This is existing practice.
* editheader extension: The implementation of header modifications is
heavily updated. Although the functionality has not changed, the
underlying code was updated to address several static analysis
warnings, runtime integer arithmetic warnings (Clang), and to match
updates in the Dovecot stream API.
+ variables extension: Made the maximum scope and variable size
configurable.
+ subaddress: Support multiple recipient_delimiters.
- enotify extension: mailto method: Fixed parsing of mailto URI with
only a header part.
- enotify plugin: mailto method: Make sure the "From:" header is set to
a usable address and not "(null)".
- Fixed writing address headers to outgoing messages. Sometimes headers
were MIME-encoded twice, yielding invalid results.
Some of the larger changes:
* Various setting changes, see https://wiki2.dovecot.org/Upgrading/2.3
* Logging rewrite started: Logging is now based on hierarchical events.
This makes it possible to do various things, like: 1) giving
consistent log prefixes, 2) enabling debug logging with finer
granularity, 3) provide logs in more machine readable formats
(e.g. json). Everything isn't finished yet, especially a lot of the
old logging code still needs to be translated to the new way.
* Statistics rewrite started: Stats are now based on (log) events.
It's possible to gather statistics about any event that is logged.
See http://wiki2.dovecot.org/Statistics for details
* ssl_dh setting replaces the old generated ssl-parameters.dat
* IMAP: When BINARY FETCH finds a broken mails, send [PARSE] error
instead of [UNKNOWNCTE]
* Linux: core dumping via PR_SET_DUMPABLE is no longer enabled by
default due to potential security reasons (found by cPanel Security
Team).
+ Added support for SMTP submission proxy server, which includes
support for BURL and CHUNKING extension.
+ LMTP rewrite. Supports now CHUNKING extension and mixing of
local/proxy recipients.
+ auth: Support libsodium to add support for ARGON2I and ARGON2ID
password schemes.
+ auth: Support BLF-CRYPT password scheme in all platforms
+ auth: Added LUA scripting support for passdb/userdb.
See https://wiki2.dovecot.org/AuthDatabase/Lua
- Input streams are more reliable now when there are errors or when
the maximum buffer size is reached. Previously in some situations
this could have caused Dovecot to try to read already freed memory.
- Output streams weren't previously handling failures when writing a
trailer at the end of the stream. This mainly affected encrypt and
zlib compress ostreams, which could have silently written truncated
files if the last write happened to fail (which shouldn't normally
have ever happened).
- virtual plugin: Fixed panic when fetching mails from virtual
mailboxes with IMAP BINARY extension.
- doveadm-server: Fix potential hangs with SSL connections
- doveadm proxy: Reading commands' output from v2.2.33+ servers could
have caused the output to be corrupted or caused a crash.
- Many other smaller fixes
Unsorted entries in PLIST files have generated a pkglint warning for at
least 12 years. Somewhat more recently, pkglint has learned to sort
PLIST files automatically. Since pkglint 5.4.23, the sorting is only
done in obvious, simple cases. These have been applied by running:
pkglint -Cnone,PLIST -Wnone,plist-sort -r -F
This has been a pkglint warning for several years now, and pkglint can even
fix it automatically. And it did for this commit.
Only in lang/mercury, two passes of autofixing were necessary because there
were nested variables.
version 3.005: Fri 22 Dec 09:43:45 CET 2017
Fixes:
- repair loose dependency on Mail::Transport [cpantesters]
version 3.004: Thu 21 Dec 09:08:52 CET 2017
Fixes:
- field unfold replaces leading whitespace into blank. [Mark Nienberg]
Improvements:
- improve docs on $msg->send().
The previous release was the last one supporting autotools,
so switch to autosetup build. Adapt options.
2017-12-15 Richard Russon <rich@flatcap.org>
* Bug Fixes
- Fix some regressions in the previous release
