upstream changes:
-----------------
* Fixed issues with DKIM and ARC verification
It was possible for some DKIM checks to fail where multiple signatures are
present due to a canonicalisation bug. This issue has now been fixed. Arc
plugin has also been fixed to support certain CV values.
* Added support for S/MIME containers
From this version, Rspamd supports .p7 containers and extracting signed
parts during the checks. For details see the following issue.
* Several important rules rework
Anton Yuzhaninov has reworked many old rules in Rspamd improving their
quality and has removed several outdated rules as well.
* Support of caching for regexp multimaps
Regexp maps can now be cached on disk which should improve loading speed of
large maps on reload/restart of Rspamd if they are unchanged.
* Neural plugin offline learning
In this mode, Rspamd can train neural network from Clickhouse so it is
possible to define better training conditions and manage learning for large
systems with more fine grained control. Please refer to the corresponding
documentation section for more details. Thanks to Andrew Lewis for
implementing this functionality.
* Other changes
Here is the list of the important changes:
[Conf] Add R_DKIM_PERMFAIL to the metric
[CritFix] Dkim: Fix simple canonicalisation if multiple signatures are presented
[CritFix] Fix controller paths normalisation
[Feature] Add INVALID_DATE rule
[Feature] Add controller endpoint for training neural
[Feature] Add sanity checks for actions thresholds
[Feature] Add support of ‘==’ and ‘!=’ in Rspamd expressions
[Feature] Composites: Improve composite atoms parser
[Feature] Docker: use Debian slim variant
[Feature] Elastic: Add some missing fields
[Feature] Extract text from img alt attributes
[Feature] Improve charset detection logic
[Feature] Lua_clickhouse: Add optional row callback for large selections
[Feature] Lua_dns_resolver: Add idna_convert_utf8 method
[Feature] Lua_mime: Add ability to do multipattern replacement
[Feature] Lua_trie: Allow to report start of the match
[Feature] Multimap: support adding map values as extra options
[Feature] Neural: Move PCA learning to a subprocess
[Feature] RBL: support matching content/image URLs only
[Feature] RBL: support use of multiple selectors
[Feature] Reputation: Allow to specify ip masks
[Feature] Support SMIME signed messages container
[Feature] Support multiple conditions for symbols
[Feature] Support ping in milter mode
[Feature] Support rspamd_text in selector regexps
[Feature] Use own daemonization routine
[Feature] Vadesecure: Implement settings_outbound feature as recommended by Vade
[Feature] rspamadm clickhouse command
[Feature] allow hyperscan for aarch64
[Fix] Allow to set priorities between post init scripts
[Fix] Allow to use maps for strings that are not zero terminated
[Fix] Apply max_lua_urls limit for emails as well
[Fix] Arc: Fix CV check on signing
[Fix] Arc: Fix signing of the broken ARC chains
[Fix] Clickhouse: escape carriage return
[Fix] Composites: Allow partial match
[Fix] Deduct type of a table methods
[Fix] Do not load errored hyperscan database
[Fix] Do not process links in ignored html tags
[Fix] Fix ClamAV result for cached encrypted file (#3395)
[Fix] Fix canonicalisation when l= tag is presented
[Fix] Fix flag shift
[Fix] Fix handling of skip/skip_process http flags
[Fix] Fix html attachments checks
[Fix] Fix issue with pushing binary formats to Lua strings
[Fix] Fix logging for rspamadm
[Fix] Fix off-by-one with init check
[Fix] Fix parsing of escape characters in quoted pairs
[Fix] Fix pushing ucl strings with \0 inside
[Fix] Fix quoted-printable soft newlines bugged case
[Fix] Fix settings in case actions are set to null (#3415)
[Fix] Fix several issues with auth results producing
[Fix] Fix smtp comments exclusion
[Fix] Fix smtp date syntax definition
[Fix] Fix substring search in case if srchlen == inlen
[Fix] Fix text selectors
[Fix] Honour systemd setting when logging to console (#3514)
[Fix] Html: Add entities collisions prevention logic (e.g. for mathml entities)
[Fix] Lua_auth_results: Quote potentially bad values in AR header
[Fix] Multimap: Fix flags usage
[Fix] Multimap: Fix scoring for combined maps
[Fix] Plug GList * leak in redis pool
[Fix] RBL: allow for multiple matches of the same label if types are different
[Fix] Rely on libev checks for file maps
[Fix] Restore simple dkim canonicalisation mode
[Fix] Return MimeCharset as we work with emails…
[Fix] Spamassassin: Fix pcre_only flags
[Fix] Spamassassin: Preserve ‘pcre_only’ flag when dealing with regexp replacements
[Fix] Try to fix GError leak
[Fix] Try to fix a mess with settings loading by adding priorities
[Fix] Try to move setings initialisation to a later stage
[Fix] Use dup fd in milter handler to avoid races with the proxy
[Fix] Use message pointer to avoid obsolete data to be cached
[Project] Rbl: Migrate to checks
[Project] Rbl: Move config code outside of the plugin
[Project] Ressurect empty prefilters as connection filters
[Project] Support connection filters registration from Lua
[Rework] Add final cleanup logic
[Rework] Add preliminary support of hyperscan caching for re maps
[Rework] Add stale cache removal
[Rework] Clickhouse: Improve performance
[Rework] Distinguish between strict config test mode
[Rework] Furhter logging improvements
[Rework] Milter_headers: improve extended_headers_rcpt support
[Rework] Move parsers to a separate lua library
[Rework] Neural: Skip composite symbols
[Rework] Rbl: Rework defaults logic
[Rework] Some tunes to cache saving
[Rework] Track maps origins
[Rework] Use full crypto hash for regexp maps
[Rules] Remove broken rule
upstream changes:
-----------------
2.6: 30 Sep 2020
* [Conf] Add missing symbols
* [Conf] Add missing symbols
* [Conf] Fix fat-fingers typo
* [Conf] Fix wrong comment in options.inc
* [Conf] Neural: Fix the default name for max_trains
* [Conf] Register a known symbol
* [Conf] Spf: Add R_SPF_PERMFAIL symbol
* [CritFix] Arc: Fix ARC validation for chains of signatures
* [CritFix] Distinguish socketpairs between different fuzzy workers
* [CritFix] Fix IDNA dots parsing
* [CritFix] Fix test assertion method
* [CritFix] Fix usage of crypto_sign it should be crypto_sign_detached!
* [Feature] Add BOUNCE rule
* [Feature] Add controller plugins support and selectors plugin
* [Feature] Add maps query method
* [Feature] Add minimal delay to fuzzy storage
* [Feature] Add multiple base32 alphabets for decoding
* [Feature] Add preliminary support of BCH addresses
* [Feature] Add query_specific endpoint
* [Feature] Allow multiple base32 encodings in Lua API
* [Feature] Allow to specify nonces manually
* [Feature] Controller: Allow to pass query arguments to the lua webui plugins
* [Feature] Fuzzy_check: Add gen_hashes command
* [Feature] Fuzzy_check: Add weight_threshold option for fuzzy rules
* [Feature] Implement address retry on connection failure
* [Feature] Improve limits in pdf scanning
* [Feature] Initial support of subscribe command in lua_redis
* [Feature] Lua_cryptobox: Add secretbox API
* [Feature] Lua_text: Add encoding methods
* [Feature] Milter_headers: Allow to activate routines via users settings
* [Feature] PDF: Add timeouts for expensive operations
* [Feature] Preliminary maps addon for controller
* [Feature] Split pdf processing object and output object to allow GC
* [Feature] Support BLIS blas library
* [Feature] Support input vectorisation by recvmmsg call
* [Feature] Support multiple base32 alphabets
* [Feature] add queueid, uid, messageid and specific symbols to selectors [Minor] use only selectors to fill vars in force_actions message
* [Feature] allow variables in force_actions messages
* [Feature] extend lua api
* [Fix] #3249
* [Fix] Allow to adjust neurons in the hidden layer
* [Fix] Another try to fix email names parsing
* [Fix] Arc: Allow to reuse authentication results when doing multi-stage signing
* [Fix] Arc: Fix bug with arc chains verification where i>1
* [Fix] Arc: Sort headers by their i= value
* [Fix] Change neural plugin's loss function
* [Fix] Deal with double eqsigns when decoding headers
* [Fix] Default ANN names in clickhouse
* [Fix] Disable reuseport for TCP sockets as it causes too many troubles
* [Fix] Disable text detection heuristics for encrypted parts
* [Fix] Distinguish DKIM keys by md5
* [Fix] Distinguish type from flags in register_symbol
* [Fix] Dmarc: Unbreak reporting after cf2ae3292ac93da8b6e0624b48a62828a51803c9
* [Fix] Do not flag pre-result of virus scanners as least if action is reject
* [Fix] Do not use GC64 workaround on 32bit platforms, omg
* [Fix] Exclude damaged urls from html parser
* [Fix] Fix FREEMAIL_REPLYTO_NEQ_FROM_DOM
* [Fix] Fix FROM_NEQ_ENVFROM
* [Fix] Fix FWD_GOOGLE rule (#1815)
* [Fix] Fix adding of the empty archive file for gzip
* [Fix] Fix aliases in forged recipients and limit number of iterations
* [Fix] Fix authentication results insertion
* [Fix] Fix calling of methods in selectors
* [Fix] Fix clen length for hiredis...
* [Fix] Fix endless loop if broken arc chain has been found
* [Fix] Fix false - operation
* [Fix] Fix get_urls table invocation
* [Fix] Fix group based composites
* [Fix] Fix headers passing in rspamd_proxy
* [Fix] Fix incomplete utf8 sequences handling
* [Fix] Fix lua_next invocation
* [Fix] Fix lua_parse_symbol_type function logic
* [Fix] Fix multiple listen configuration
* [Fix] Fix occasional encryption of the cached data
* [Fix] Fix parsing boundaries with spaces
* [Fix] Fix passing of methods arguments
* [Fix] Fix poor man allocator algorithm
* [Fix] Fix regexp selector and add flattening
* [Fix] Fix rfc base32 encode ordering (skip inverse bits)
* [Fix] Fix rfc based base32 decoding
* [Fix] Fix sockets leak in the client
* [Fix] Fix storing of the original smtp from
* [Fix] Fix types check and types usage in lua_cryptobox
* [Fix] Fix unused results
* [Fix] Fuzzy_check: Disable shingles for short texts (really)
* [Fix] Ical: Fix identation grammar
* [Fix] Improve part:is_attachment logic
* [Fix] Mmap return value must be checked versus MAP_FAILED
* [Fix] One more fix to skip images that are not urls
* [Fix] Pdf: Support some weird objects with no newline before endobj
* [Fix] Rbl: Fix ignore_defaults in conjunction with ignore_whitelists
* [Fix] Restore support for `for` and `id` parts in received headers
* [Fix] Segmentation fault in contrib/lua-lpeg/lpvm.c on ppc64el
* [Fix] Skip spaces at the boundary end
* [Fix] Slashing fix: fix captures matching API
* [Fix] Spamassassin: Rework metas processing
* [Fix] Store reference of upstream list in upstreams objects
* [Fix] Understand utf8 in content-disposition parser
* [Fix] Unify selectors digest functions
* [Fix] Use `abs` value when checking composites
* [Fix] Use strict IDNA for utf8 DNS names + add sanity checks for DNS names
* [Fix] Use unsigned char and better support of utf8 in ragel parser
* [Fix] add missing selector_cache declaration
* [Project] Add `L` flag for regexps to save start of the match in Hyperscan
* [Project] Add `lower` method to lua_text
* [Project] Add a simple matrix Lua library
* [Project] Add implicit bitcoincash prefix
* [Project] Add linalg ffi library for prototyping
* [Project] Add methods to append data to fuzzy requests
* [Project] Add routine to call a generic lua function
* [Project] Add ssyev method interface
* [Project] Add tensors index method
* [Project] Add text:sub method
* [Project] Allow rspamd_text based selectors
* [Project] Allow to specify re_conditions for regular expressions
* [Project] Attach extensions to the binary fuzzy commands
* [Project] Bitcoin: BTC cash addresses needs some checksum validation
* [Project] Cleanup the redis script
* [Project] Convert bitcoin rules to the new regexp conditions feature
* [Project] Detect memrchr in systems that supports it
* [Project] Do not listen sockets in the main process
* [Project] Implement 'probabilistic' learn mode for ANN
* [Project] Implement BTC polymod in C as it requires 64 bit ops
* [Project] Implement bitcoin cash validation in a proper way
* [Project] Implement extensions logic for fuzzy storage
* [Project] Implement symbols insertion in multiple results mode
* [Project] Lua_text: Add method memchr
* [Project] Neural: Add PCA loading logic
* [Project] Neural: Fix PCA based learning
* [Project] Neural: Fix matrix gemm
* [Project] Neural: Further PCA fixes
* [Project] Neural: Implement PCA in learning
* [Project] Neural: Implement PCA learning
* [Project] Neural: Implement PCA on ANN forward
* [Project] Neural: Implement PCA serialisation
* [Project] Neural: Start PCA implementation
* [Project] Neural: Use C version of scatter matrix producing
* [Project] Preliminary support of lua conditions for regexps
* [Project] Preliminary usage of the reuseport
* [Project] Process composites separately for each shadow result
* [Project] Remove old code
* [Project] Rework scan result functions to support shadow results
* [Project] Rework some more functions to work with shadow results
* [Project] Some more fixes
* [Project] Start results chain implementation
* [Project] Support fun iterators on rspamd_text objects
* [Project] Support multiply, minus and divide operators in expressions
* [Project] Tensor: Move scatter matrix calculation to C
* [Rework] Allow to specify exat metric result when adding a symbol
* [Rework] Change and improve openblas detection and usage
* [Rework] Close listen sockets in main after fork
* [Rework] Further rework of lua urls extraction API
* [Rework] Lua_cryptobox: Allow to store output of the hash function
* [Rework] Lua_task: Add more methods to deal with shadow results
* [Rework] Modernize logging for expressions
* [Rework] Remove empty prefilters feature - we are not prepared...
* [Rework] Remove old FindLua module, disable lua fallback when LuaJIT is enabled
* [Rework] Rework and refactor forged recipients plugin
* [Rework] Rework expressions processing
* [Rework] Rework fuzzy commands processing
* [Rework] Rework url flags handling API
* [Rework] Rework urls extraction
* [Rework] Split operations processing and add more debug logs
* [Rework] Update zstd to 1.4.5
* [Rework] Use google-ced instead of libicu chardet as the former sucks
* [Rework] add alias util:parse_addr for util:parse_mail_address
* [Rework] get rid of util:parse_addr duplicating the util:parse_mail_address, replace where used
* [Rules] Allow prefix for bitcoin cash addresses
* [Rules] More fixes for bitcoin cash addresses decoding
* [Rules] Refactor bleach32 addresses handling
correct install_name_tool -id on macOS, where this fixes CHECK_SHLIBS
(and probably runtime behavior too). While here, the patch to link with
-lrt on NetBSD has been upstreamed; remove. Bump PKGREVISION.
from GitHub user @sjorge + extra patch from me
Closes NetBSD/pkgsrc#60
2.5: 01 Apr 2020
* [Conf] Mark Rspamd emailbl as ignore whitelist
* [Conf] RBL: Add missing emails = true option
* [Feature] Add support for scripts in fuzzy storage
* [Feature] Arc: Add whitelisted_signers_map option
* [Feature] Implement hosts file processing
* [Feature] Neural: Introduce classes bias that allows non-equal classes learning
* [Feature] Update libev to 4.33
* [Fix] Another brain damage html standard adoptions
* [Fix] Another fix for brain damaged obs-fws state
* [Fix] Fix flags that caused force_actions failure
* [Fix] Fix logging issue
* [Fix] Fix lua symbols scores registration when config does not define scores
* [Fix] Fix opaque maps logic
* [Fix] Fix parsing of the html tags with no spaces after attributes
* [Fix] Fix some corner cases in urls parsing, add limits
* [Fix] Fix tlds extraction if custom composition rules are used
* [Fix] Fix variables replacement in mempool
* [Fix] Improve base64 detection
* [Fix] Normalize dynamic scores in ANN correctly
* [Fix] Plug memory leak introduced by #3153
* [Fix] Stat_redis_backend: Fix memory leak and simplify learn path
* [Fix] Try hard to deal with ghost workers
* [Fix] metadata_exporter default formatter
* [Rework] Change the way to extract URLs when dealing with alternative parts
* [Rework] Fix various url extraction issues
* [Rework] Re cache: Load compiled hyperscan in the main process as well
* [Rework] Re cache: Load hyperscan early
* [Rework] Rework URL structure: adjust tld part
* [Rework] Rework URL structure: host field
* [Rework] Rework URL structure: more structure optimisations
* [Rework] Rework URL structure: user field
* [Rework] URL: Another update for urls extraction logic
* [Rework] Urls: Improve query urls handling
* [Rework] Urls: adopt html related stuff
* [Rework] Urls: more rework of the urls sets
* [Rework] Urls: process query urls in HTML urls correctly
* [Rework] Urls: rework urls hash structure
* [Rework] Urls: update lua libraries
* [Rework] Use multiple search tries for different url extraction types
2.4: 26 Feb 2020
* [CritFix] Fix parsing of the content type attributes
* [Feature] Clickhouse: Add extra columns support
* [Feature] Rbl: Add url_compose_map option for RBL rules
* [Fix] 'R' flag is for all headers regexp
* [Fix] Allow to reset settings id from Lua (e.g. because of the priority)
* [Fix] Avoid collisions in mempool variables by changing fuzzy caching logic
* [Fix] Avoid strdup usage for symbols options
* [Fix] Do not trust stat(2) it lies
* [Fix] Filter all options for symbols to have sane characters
* [Fix] Fix all headers iteration
* [Fix] Fix allowed_settings for neural
* [Fix] Fix listen socket parsing
* [Fix] Fix maps expressions evaluation
* [Fix] Fix sentinel connections leak by using async connections
* [Fix] Fix smtp message on passthrough result
* [Fix] Fix tld compositon rules
* [Fix] Fuzzy_storage: Do not check for shingles if a direct hash has been found
* [Fix] Lua_mime: Do not perform QP encoding for 7bit parts
* [Fix] Neural: Distinguish missing symbols from symbols with low scores
* [Fix] Support listening on systemd sockets by name
* [Project] Add lua_urls_compose library
* [Project] Allow to set a custom log function to the logger
* [Project] CDB maps: Start making cdb a first class citizen
* [Project] Clickhouse: Add extra columns concept
* [Project] Fix urls composition rules, add unit tests
* [Project] Unify cdb maps
* [Rework] Logger infrastructure rework
* [Rework] Refactor libraries structure
* [Rework] Rework SSL caching
* [Rework] Update snowball stemmer to 2.0 and remove all crap aside of UTF8
upstream changes:
-----------------
2.3: 04 Feb 2020
* [Conf] SPF is no longer a C module
* [Conf] Update spamtrap map path example
* [CritFix] Fix html entities decoding
* [CritFix] Fix re cache when mix of pcre and hyperscan is used
* [Feature] Allow milter code to deal with multiple headers
* [Feature] Antivirus: Add avast support
* [Feature] Dkim_signing: Allow to sign via milter_headers
* [Feature] Implement content hashes
* [Feature] Lua_text: Add regexp split iterator method
* [Feature] Lua_text: Implement flattening of the input tables
* [Feature] Send quit command to Redis
* [Feature] Speed up is_ascii function
* [Feature] Spf: Add external_relay option
* [Fix] Avoid double escaping
* [Fix] Fix O(N^2) algorithm
* [Fix] Fix arc seal validation
* [Fix] Fix base tag processing according to stupid HTML renderer behaviour
* [Fix] Fix dealing with `\0` in ucl strings and JSON
* [Fix] Fix gpg parts misdetection
* [Fix] Fix ignored symbols exporting
* [Fix] Fix processing of numeric url's
* [Fix] Fix processing of the closed tcp connections
* [Fix] Fix regexp type check for pcre2
* [Fix] Fix urls encode function
* [Fix] Fix urls shifting when doing decode to include separators
* [Fix] Fix white on white rule and add is_leaf flag
* [Fix] Further fixes in charset detection
* [Fix] Ignore diacritics in chartable module for specific languages
* [Fix] Limit size of symbols options by max_opts_len option
* [Fix] More fixes in html tag content calculations
* [Fix] Plug memory leak in fuzzy storage
* [Fix] Process high priority settings even if settings/id has been specified
* [Fix] Select a different upstream on last retransmit
* [Fix] Treat soft hyphen as zero width space
* [Fix] Try harder to watch the lifetime of the key_stat
* [Fix] Use ipv6-mapped-ipv4 addresses in radix trie
* [Project] Add logic to break execution when processing symbols*
* [Project] Add methods to set specific content for mime parts from Lua
* [Project] Lua_content: support PDF files
* [Project] Move dns_tool to using of the rspamd_spf from FFI module
* [Project] Preliminary SPF plugin in Lua
* [Project] Show debug stat for memory pool
* [Project] Some rework about specific data that is now tagged
* [Project] Start reworking of the mempool structure
* [Rework] Allow to add userdata as symbols options
* [Rework] Change mime part specifics handling
* [Rework] Move LRU SPF cache from spf plugin
* [Rework] Rework HTML tags content attachment
* [Rework] Rework options hash structure
* [Rework] Start lua_content library
* [Rework] Stop using of uthash for http headers
* [Rework] Use faster hashing approach for memory pools variables
* [Rules] Add PDF related rules
upstream changes:
-----------------
[Conf] Antivirus: Fix the default config
[Feature] Add verdict library in lua
[Feature] Allow exception when choosing upstream
[Feature] Allow to disable symbols from the metric config
[Feature] Allow to limit maps per specific worker
[Feature] Always validate Rspamd protocol output
[Feature] Antivirus: Add preliminary virustotal support
[Feature] Clickhouse: Rework Clickhouse collection logic
[Feature] Improve base64 usage
[Feature] Shutdown timeout is now associated with task timeout
[Fix] #3129 Multiple classifiers on redis working incorrectly
[Fix] Allow real upstreams configuration
[Fix] Another try to fix slow callbacks and timers
[Fix] Check results of write message as SSL can bork them
[Fix] Clickhouse: Avoid potential races in collection
[Fix] Clickhouse: Fix periodic script
[Fix] Fail DNS upstream on each retransmit attempt
[Fix] Fix consistent hashing when upstreams are marked inactive
[Fix] Fix issues found
[Fix] Fix off-by-one in retries for the proxy
[Fix] Fix termination
[Fix] Fix upstreams exclusion logic
[Fix] Fix utf8 validation for symbols options and empty strings
[Fix] Oops, fix maps reload
[Fix] Rbl: Allow utf8 lookups for IDN domains
[Fix] Sigh, another try to fix brain-damaged openssl
[Project] Add fast utf8 validation library
[Project] Use own utf8 validation instead of glib
[Rework] Another phase of finish actions rework
[Rework] Further cmake system rework
[Rework] Further isolation of the controller’s functions
[Rework] Make cmake structure more modular
[Rework] Move cmake modules to a dedicated path
[Rework] Replace controller functions by any scanner worker if needed
[Rework] Rework final scripts logic
[Rework] Rewrite rspamd_str_make_utf_valid function
pkgsrc changes:
---------------
* Change BUILDLINK_TRANSFORM to BUILDLINK_FNAME_TRANSFORM to appease
pkglint.
* Add c++ to USE_LANGUAGES because the configure step failed.
upstream changes:
-----------------
2.1: 28 Oct 2019
* [Conf] Update neural.conf
* [CritFix] Fix dkim verification for multiple headers listed
* [Feature] Add support of uudecode
* [Feature] Allow to explicitly set events backend
* [Feature] Implement configurable limits for SPF lookups
* [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic
* [Feature] Multimap: Do not check files in office archives
* [Feature] Neural: Add sampling when storing training vectors
* [Feature] SPF: Allow to disable AAAA checks in configuration
* [Feature] Spf: Add limits configuration support
* [Feature] Store etag in cached HTTP maps + better logging
* [Feature] Support segwit BTC addresses, fix LTC verification
* [Feature] Support uuencoding
* [Fix] Add configurable number of threads for OpenBLAS
* [Fix] Add workaround for ragel 7 in hyperscan related maps code
* [Fix] Another fix for numeric urls parsing
* [Fix] Correct EMA time calculations
* [Fix] Do not treat archives as text
* [Fix] Do not use strdup on data extracted from lua
* [Fix] Fix a failure calcuating URL reputation.
* [Fix] Fix crash due to constructors init order
* [Fix] Fix crash on parts with no cd
* [Fix] Fix empty prefilters that require mime structures
* [Fix] Fix event loop creation
* [Fix] Fix issues sending DMARC reports.
* [Fix] Fix misprint
* [Fix] Fix saving of the file maps
* [Fix] Fix size calculations when converting from utf16
* [Fix] Fix support of disable_monitoring in rbl
* [Fix] Fix use-after-free
* [Fix] Fix zip files check to relax requirements
* [Fix] Important hiredis fixes
* [Fix] Lot's of fixes in maps check logic
* [Fix] Lua_tcp: Deal with temporary fails on write
* [Fix] Lua_tcp: Make write errors fatal and rework error handlers
* [Fix] Meta: Filter some more values
* [Fix] Neural: Add protection agains infinities
* [Fix] Oops, fix math.huge invocation
* [Fix] Plug memory leak
* [Fix] Sigh, another email to string fix
* [Fix] Try to fix another ownership race in ssl connection
* [Fix] Uuencode: Fix parsing of corrupted uuencode
* [Fix] lua_scanners - razor rename need_check function
* [Rework] Require CMake 3.9 to work, remove manual lto crap
2.0: 11 Oct 2019
* [Conf] Add BROKEN_HEADERS_MAILLIST composite
* [Conf] Add path to greylist-whitelist-domains.inc
* [Conf] Clarify documentation in the config files
* [Conf] Introduce maps.d directories
* [Conf] Log settings id by default
* [Conf] Make LEAKED_PASSWORD_SCAM a composite rule again
* [Conf] Move all surbl/emails rules to rbl
* [Conf] Register new Spamhaus codes
* [Conf] Remove configs for deleted modules
* [Conf] Remove surbl parts, fix hash_format attribute
* [Conf] Show autolearn sample
* [Conf] Slashing: Change default stats backend to Redis
* [Conf] Surbl: Utilise new `check_emails` option
* [Conf] Update header
* [Conf] Use multi-prefixes RBLs in the default config
* [CritFix] Deal with case-sensivity in Content-Disposition parser
* [CritFix] Eliminate old endpoint
* [CritFix] Fix case sensivity when parsing Content-Type
* [CritFix] Fix loading of DKIM public keys
* [CritFix] Fix procesing of urls
* [CritFix] Fix whitelisting when both spf and dkim are required to be valid
* [CritFix] Langdet: Fix language detection where no stop words found
* [Feature] Add description to the groups
* [Feature] Add limit for number of URLs in Lua
* [Feature] Add logging of groups to the log_format
* [Feature] Add lua_smtp library
* [Feature] Add maps cache and type refinement
* [Feature] Add p0f scanner
* [Feature] Adopt emails module to use lua_maps
* [Feature] Allow options matching in composites
* [Feature] Allow selectors in rbl module
* [Feature] Allow to output group results
* [Feature] Asn: Allow to use bgpdump when NET::MRT is broken
* [Feature] Calculate tokens occurrences distribution
* [Feature] Clickhouse: Add authenticated user and settings id columns
* [Feature] Clickhouse: Store groups data
* [Feature] Clickhouse: Utilise LowCardinality feature
* [Feature] Implement Redis prefixes registration logic
* [Feature] Implement settings id propagation between deps
* [Feature] Improve AV results caching
* [Feature] Improve autolearning
* [Feature] Improve logging locking logic (remove it actually)
* [Feature] Improve settings processing
* [Feature] Langdet: Limit number of stop words to be checked
* [Feature] Libucl: Allow to sort keys in ucl objects
* [Feature] Lua_config: Extend get symbols method
* [Feature] Lua_maps: Allow static maps for key-value pairs
* [Feature] Lua_mimepart: Add function filter_words
* [Feature] Lua_selectors: Add `words` selector
* [Feature] Lua_selectors: Add sort and uniq transform functions
* [Feature] Lua_selectors: Allow table arguments for selectors
* [Feature] Lua_tcp: Add preliminary support of SSL connections
* [Feature] Lua_trie: More flexible API
* [Feature] Lua_util: Add filter_specific_url function
* [Feature] Lua_util: table_digest can now recursively traverse tables
* [Feature] Maillist: Improve detection
* [Feature] Maps: Allow caching for complex maps
* [Feature] Monitored: Support random lookups
* [Feature] Multimap: Add combined maps prototype
* [Feature] Multimap: Add dependend maps via redis keys selectors
* [Feature] Multimap: Allow multiple email addresses matches
* [Feature] Multimap: Also check detected charset when do filename checks
* [Feature] Output number of messages processed to proctitle
* [Feature] Perform clean SSL shutdown
* [Feature] Performance: Do not use base64 SIMD version for bad inputs
* [Feature] RBL: Support bit results in replies
* [Feature] RBL: Support type specific prefixes
* [Feature] Ratelimit: Consider number of SMTP recipients
* [Feature] Rbl: Add ability to check urls
* [Feature] Rbl: Add resolve_ip based RBLs
* [Feature] Rbl: Make config checks much more strict
* [Feature] Rbl: Support per-rule whitelists
* [Feature] Rbl: Support process script
* [Feature] Rbl: Support replyto addresses
* [Feature] SURBL: Allow to check email domains
* [Feature] Selectors: Add `list` generator
* [Feature] Selectors: Add `specific_urls` extractor
* [Feature] Selectors: Add flatten function
* [Feature] Selectors: Support filter_map and apply_map functions
* [Feature] Store Clickhouse data outside of lua alloc
* [Feature] Support caching for encrypted files and macros
* [Feature] Support images when extracting urls
* [Feature] Support more hyperscan flags
* [Feature] Support protocol flags
* [Feature] URL: Apply stringprep to hostnames to filter garbage
* [Feature] Upstreams: Add lazy resolving logic to all upstreams
* [Feature] Upstreams: Set noresolve flag on numeric upstreams
* [Feature] Use `scores` in apply section
* [Feature] Use maps logic from lua_maps for multimap
* [Feature] Use random monitored in rbl module
* [Feature] lua_scanners - add Razor support
* [Fix] Add another safe-guard in urls processing
* [Fix] Add debug to ssl, fixed write hangs
* [Fix] Add missing groups to C callback symbols
* [Fix] Add more checks for ghosts symbols
* [Fix] Allow to enable or add new actions via settings
* [Fix] Allow to set 0 size for spf/dkim caches
* [Fix] Another bunch of fixes towards protocol mess
* [Fix] Another fix to deal with bad URLs
* [Fix] Arc: Another bunch of fixes for arc signing
* [Fix] Arc: More arc signing fixes
* [Fix] Avoid another overflow in fpconv
* [Fix] Clickhouse: Fix quoting
* [Fix] Clickhouse: Fix retention query quoting
* [Fix] Distinguish empty and non-empty prefilters
* [Fix] Distinguish remote and local addrs parsing
* [Fix] Do not assert if length of sig is bad, just fail verification
* [Fix] Do not assert if we have broken mime boundary in the headers
* [Fix] Do not call implicit strlen to avoid issues
* [Fix] Do not count images urls when checking url regexps for compatibility
* [Fix] Do not output rbl suffix in symbol option
* [Fix] Do not use config pool to avoid issues with double reload
* [Fix] Do not use ephemeral string
* [Fix] Do not use lightuserdata for traceback
* [Fix] Do not use priority in metric registration
* [Fix] Emails: Check email sanity before testing on BL
* [Fix] Emails: Fix misprint in key name
* [Fix] Escape utf in regexp to dodge ragel/hyperscan issue
* [Fix] Extend task_timeout to postfilters stage
* [Fix] Fix ARC signing after fixing another bug in it...
* [Fix] Fix AV scan logic
* [Fix] Fix DMARC_NA behaviour in case of no valid policies
* [Fix] Fix LRU hash iteration logic
* [Fix] Fix alignment mess
* [Fix] Fix configuring symbols without scores
* [Fix] Fix disabling of the actions
* [Fix] Fix dkim signing exceptions
* [Fix] Fix embedded images linking logic
* [Fix] Fix events leak
* [Fix] Fix eviction corner case
* [Fix] Fix fuzzy image score calculation #2962
* [Fix] Fix hang in fuzzy_learn when explicit rotation is set
* [Fix] Fix headers propagation logic
* [Fix] Fix hearbeats restart issue
* [Fix] Fix history reset
* [Fix] Fix log parameter
* [Fix] Fix lua_ip_equal logic
* [Fix] Fix more issues with nested messages + tests
* [Fix] Fix normalization of non-alphabet based languages
* [Fix] Fix offsets when parsing message/rfc822 in multipart
* [Fix] Fix options in rbl symbols
* [Fix] Fix out of bound access in lua logger
* [Fix] Fix out-of-bound read in qp decode
* [Fix] Fix parent CTE propagation
* [Fix] Fix parsing of the received headers with empty part
* [Fix] Fix pending checks for events
* [Fix] Fix printing of NULL pointer with fixed length
* [Fix] Fix race condition in watcher handler
* [Fix] Fix read-after-end in quoted printable decoding
* [Fix] Fix redis sentinel support
* [Fix] Fix registry leak in case of DNS errors
* [Fix] Fix reload logic
* [Fix] Fix sending of large entries via HTTPS
* [Fix] Fix settings reload
* [Fix] Fix some more corner cases for fpconv
* [Fix] Fix trie code when there are regexps and Hyperscan is absent
* [Fix] Further fixes to printing of the FP numbers
* [Fix] Fuzzy_check: Fix timeouts
* [Fix] Grrr, fix empty ip case
* [Fix] Html: Fix processing of fjlig entity
* [Fix] Lang_det: Try better to distinguish Chinese and Japanese
* [Fix] Lua_mime: Fix reversed extensions map
* [Fix] Lua_task: Fix message-less API
* [Fix] Lua_tcp: Report connection failures
* [Fix] Lua_tcp: Various fixes and debugging improvements
* [Fix] Metadata_exporter: This plugin is idempotent not a postfilter
* [Fix] More fixes to extract_specific_urls
* [Fix] More stages fixes
* [Fix] Neural: Another bunch of fixes
* [Fix] Neural: use version in ANN key profile
* [Fix] Postpone lua state destruction to allow lua dtors to be used
* [Fix] Prefer surbl/emails rule on rbl to preserve compatibility
* [Fix] RBL: Fix behaviour of emails_domainonly
* [Fix] Ratelimit: Fix dynamic score
* [Fix] Rbl: Fix emailbl functions
* [Fix] Really fix hyperscan workaround
* [Fix] Set sanity limits for pcre2
* [Fix] Settings: Fix settings check flags
* [Fix] Sort keys when getting data from Lua when filling rules
* [Fix] Statistics: Do not query Redis tokens when there are no learns
* [Fix] Stop IO event on write finished in http connection
* [Fix] Use heuristically detected text parts data
* [Fix] Various fixes to QP encoding algorithm
* [Fix] Various fixes to SSL state machine handler
* [Fix] Various fixes to asn module
* [Fix] Workaround for empty charset in rfc2231 encoding
* [Project] Switch from torch to KANN
* [Project] Add heartbeat events
* [Project] Add preliminary support of the Kaspersky Scan Engine
* [Project] Add preliminary version of maps expressions
* [Project] Add preprocessed settings to the config structure
* [Project] Add simple forward propagation function
* [Project] Add small helpers for migration simplifications
* [Project] Allow to replace body in milter
* [Project] Bundle libev
* [Project] First refactoring step libevent->libev
* [Project] Implement syntax highlighting for Lua
* [Project] Lua_magic: Adopt lua_magic stuff in mime_types
* [Project] Remove libfann, gd and other unsupported stuff
* [Project] Remove torch
* [Project] Rework upstreams
* [Rework] Allow execution of async events when hs compiles regexps
* [Rework] Bayes expiry: eliminate `default` expiration mode
* [Rework] Dkim: Remove signing code
* [Rework] Dkim_signing: Move sign condition to dkim_signing
* [Rework] Do not lowercase all data send to ClickHouse
* [Rework] Drop url tags
* [Rework] Eliminate lua_squeeze as it has shown no improvements
* [Rework] Eliminate virtual scan time as it is useless
* [Rework] Lua core: Use lightuserdata to index classes
* [Rework] Lua_util: Another rework for extract_specific_urls
* [Rework] Migrate from ip_score to reputation
* [Rework] Move mime modification functions to lua_mime library
* [Rework] Rbl: Major whitelisting logic rework
* [Rework] Remove deprecated plugins
* [Rework] Remove log helper worker
* [Rework] Remove rspamd.classifiers.lua
* [Rework] Rename filter.h to a more sane name
* [Rework] Reorganise selectors implementation
* [Rework] Replace linenoise with replxx
* [Rework] Reputation: Remove ipnet from the ip reputation
* [Rework] Reputation: Slashing - change name of symbols
* [Rework] Rework children operations
* [Rework] Rework config reload
* [Rework] Rework expression API
* [Rework] Rework image urls processing
* [Rework] Rework initialisation to reduce static leaks count
* [Rework] Rework request headers processing
* [Rework] Slashing: Change versioning schema - move to 2.0
* [Rework] Slashing: Turn off postfilters when passthrough result is set
* [Rework] Start moving to replxx
* [Rework] Stop support of signed HTTP maps to simplify code
* [Rework] Store ASN as UInt32 in ClickHouse
* [Rework] Url_redirector: Rewrite plugin
* [Rework] Use a dedicated library for autolearn
* [Rework] Use libsodium instead of hand crafted crypto implementations
* [Rework] Use opaque structure to store a table of mime headers
* [Rules] Add dedicated bitcoin addresses filter rule
* [Rules] Add more detection to LEAKED_PASSWORD_SCAM
* [Rules] Catch LTC addresses
* [Rules] Reduce weight of RSPAMD_EMAILBL
* [Rules] Rework LEAKED_PASSWORD_SCAM rule one more time
1.9.4: 23 May 2019
* [CritFix] Fix case sensitivity when parsing Content-Type
* [Fix] Arc: Another bunch of fixes for arc signing
* [Fix] Arc: More arc signing fixes
* [Fix] Avoid another overflow in fpconv
* [Fix] Fix ARC signing after fixing another bug in it...
* [Fix] Fix dkim signing exceptions
* [Fix] Fix some more corner cases for fpconv
* [Fix] Further fixes to printing of the FP numbers
* [Fix] Ratelimit: Fix dynamic score
1.9.3: 13 May 2019
* [Conf] Add IP_SCORE_FREEMAIL composite rule
* [Feature] Add cryptobox method to generate dkim keypairs
* [Feature] Add fast hashes to lua cryptobox hash
* [Feature] Add least passthrough results
* [Feature] Allow oversign if exists mode
* [Feature] Clickhouse: Modernise table initial schema
* [Feature] Implement IUF interface for specific fast hashes
* [Feature] Lua_util: Allow to obfuscate different fields
* [Feature] Tune memory management in Rspamd and Lua
* [Fix] Avoid buffer overflow when printing long lua strings
* [Fix] Change the default oversigning headers to a more sane list
* [Fix] Clickhouse: Do not store digest as it is not needed now
* [Fix] Clickhouse: Fix lots of storage issues
* [Fix] Clickhouse: Support custom actions
* [Fix] Deny URLs where hostname is bogus
* [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
* [Fix] Fix DoS caused by bug in glib
* [Fix] Fix UCL parsing of the multiline strings
* [Fix] Fix buffer overflow when printing small floats
* [Fix] Fix init code for servers keypairs cache
* [Fix] Fix issue with urls with no tld (e.g. IP)
* [Fix] Fix memory in arc signing logic
* [Fix] Fix memory leak in language detector during reloads
* [Fix] Fix mixed case content type processing
* [Fix] Fix processing of the ip urls in file
* [Fix] Fix use after free
* [Fix] HTML: Fix `size` attribute processing
* [Fix] Hum, it seems that 99ff1c8 was not correct
* [Fix] Lua_task: Fix task:get_from method
* [Fix] Preserve fd when mapping file to scan
* [Fix] Re-use milter_headers settings when doing arc signing
* [Fix] Set dmarc force action as least action
* [Fix] Switch to GMT
* [Fix] allow PKCS7 signatures to be text/plain, too
* [Project] Add initial version of the vault management tool
* [Project] Add vault support for DKIM and ARC signing
* [Project] Implement keys rotation in the vault
* [Project] Improve dkim keys generation for vault
* [Project] Improve keys creation in rspamadm vault
* [Rework] Move lua_worker to a dedicated unit
* [WebUI] Add URL fragments (#) support
* [WebUI] Fix AJAX request URL
1.9.2: 16 Apr 2019
* [Conf] Allow to load users plugins from plugins.d
* [Conf] oversign openpgp and autocrypt headers
* [Feature] Add SPF FFI library for Lua
* [Feature] Add more verbosity for SPF caching
* [Feature] Antivirus: Handle encrypted files specially
* [Feature] Clickhouse: Slashing - add new fields to CH
* [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table
* [Feature] Dkim_signing: Allow to use new options as maps
* [Feature] Import fpconv library
* [Feature] Lua_maps: Allow static regexp and glob maps
* [Feature] Parse ical files
* [Feature] Rspamadm: Add dns_tool utility
* [Feature] Store SPF records digests
* [Feature] Use fpconv girsu2 implementation for printing floats
* [Fix] Clickhouse: Use integer seconds when inserting rows
* [Fix] Fix floating point printing
* [Fix] Fix processing of embedded urls
* [Fix] Lua_clickhouse: Fix CH errors processing
* [Fix] Make spf digest stable
* [Fix] Properly detect encrypted files in zip archives
* [Fix] Slashing: Store times in GMT timezone in ClickHouse
* [Rules] Add additional conditions to perform BTC checks
* [Rules] Fix pay-to-hash addresses validation
1.9.1: 5 Apr 2019
* [Conf] Add vendor groups for symbols
* [Feature] Add `rspamadm template` command
* [Feature] Allow to add messages from settings
* [Feature] Allow unconnected DNS servers operations
* [Feature] Check limits after being set, migrate to uint64
* [Feature] Greylist: Allow to disable greylisting depending on symbols
* [Feature] Improve lua binary strings output
* [Feature] Mime_types: Implement user configurable extension filters
* [Feature] Mime_types: When no extension defined, detect it by content
* [Feature] Preprocess config files using jinja templates
* [Feature] Replies: Filter replies sender to limit whitelisting to direct messages
* [Feature] Treat all tags with HREF as a potential hyperlinks
* [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM
* [Fix] Add crash safety for HTTP async routines
* [Fix] Another fix for Redis sentinel
* [Fix] Clickhouse: Fix table schema upload
* [Fix] Core: Fix squeezed dependencies handling for virtual symbols
* [Fix] Finally fix default parameters parsing in actions section
* [Fix] Fix ES sending logic (restore from coroutines mess)
* [Fix] Fix finishing script for clickhouse collection
* [Fix] Fix priority for regexp symbols registriation
* [Fix] Fix various issues found by PVS Studio
* [Fix] Initialize lua debugging earlier
* [Fix] Neural: Fix training
* [Fix] Rework cached Redis logic to avoid sentinels breaking
* [Fix] SURBL: Fix regression in surbl module
* [Fix] Fix double signing in the milter
* [Project] Add support of HTTP proxy in requests
* [Rework] Change lua global variables registration
* [Rework] Rework HTML content urls extraction
* [Rework] Start rework of aliasing in Rspamd
* [WebUI] Combine Scan and Learning into one tab
* [WebUI] Fix symbol score input type
* [WebUI] Show grayed out pie
* [WebUI] Update Throughput summary values dynamically
1.9.0: 12 Mar 2019
* [Conf] Add missing includes
* [Conf] Move to options
* [Conf] Rbl: DWL is actually special whitelist
* [Conf] Relax some uribl rules
* [Conf] Remove abuse.ch
* [CritFix] Html: Entities are not valid within tag params values
* [Feature] Add `rspamadm mime sign` tool
* [Feature] Add configgraph utility
* [Feature] Add dedicated ZW spaces detection for URLs
* [Feature] Add flag to url object when visible part is url_like
* [Feature] Add method task:lookup_words
* [Feature] Add pyzor support (by crosenberg)
* [Feature] Allow to add upstream watchers to Lua API
* [Feature] Allow to set rewrite subject pattern from settings
* [Feature] Better escaping of unicode
* [Feature] Clickhouse: Allow to store subject in Clickhouse
* [Feature] Core: Add QP encoding utility
* [Feature] Core: Add libmagic detection for all parts
* [Feature] Core: Add support for gzip archives
* [Feature] Core: Allow to construct scan tasks from raw data
* [Feature] Core: Detect charset in archived files
* [Feature] Core: Ignore and mark invisible spaces
* [Feature] Core: Normalise zero-width spaces in urls
* [Feature] Core: Process data urls for images
* [Feature] Core: Relax quoted-printable encoding
* [Feature] Core: Support RFC2231 encoding in headers
* [Feature] Core: Support telephone URLs
* [Feature] Core: allow to emit soft reject on task timeout
* [Feature] DCC: Add bulkness and reputation checks to dcc
* [Feature] Elastic: Modernize plugin
* [Feature] Export visible part of url to lua
* [Feature] Fuzzy_storage: add preliminary support of rate limits
* [Feature] HTML: Specially treat data urls in HTML
* [Feature] Implement event watchers for upstreams
* [Feature] Implement includes tracing in Lua
* [Feature] Improve dkim part in configwizard
* [Feature] Lua_scanners: Add VadeSecure engine support
* [Feature] Lua_task: Add flexible method to get specific urls
* [Feature] Mime_types: Add MIME_BAD_UNICODE rule
* [Feature] Mime_types: Use detected content type as well
* [Feature] Plugins: Add preliminary version of the external services plugin
* [Feature] Query sentinel on master errors
* [Feature] Regexp: Allow local lua functions in Rspamd regexp module
* [Feature] Rspamadm: Allow to append footers to plain messages
* [Feature] Rspamadm: Allow to rewrite headers in messages
* [Feature] Selectors: Add `ipmask` processor
* [Feature] Settings: Allow hostname match
* [Feature] Settings: Allow local when selecting settings
* [Feature] Settings: Allow multiple selectors
* [Feature] Settings: Allow to inverse conditions
* [Feature] Support User-Agent in HTTP requests
* [Feature] Support ed25519 dkim keys generation
* [Feature] Try to filter bad unicode types during normalisation
* [Feature] external_services - oletools (olefy) support
* [Feature] lua_scanners - icap protocol support
* [Feature] lua_scanners - spamassassin spam scanner
* [Fix] Add filter for absurdic URLs
* [Fix] Add some more cases for Received header
* [Fix] Allow to disable/enable composite symbols
* [Fix] Arc: Use a separated list of headers for arc signing
* [Fix] Archive: Final fixes for 7z archives
* [Fix] Clickhouse: Fix database usage
* [Fix] Controller: Make save stats timer persistent
* [Fix] Core: Detect encrypted rarv5 archives
* [Fix] Core: Don't detect language twice
* [Fix] Core: Fix address rotation bug
* [Fix] Core: Fix content calculations for message parts
* [Fix] Core: Fix emails comments parsing and other issues
* [Fix] Core: Fix etags support
* [Fix] Core: Fix headers folding on the last token
* [Fix] Core: Fix iso-8859-16 encoding
* [Fix] Core: Fix log_urls flag (and encrypted logging)
* [Fix] Core: Fix part length when dealing with boundaries
* [Fix] Core: Fix parts distance calculations
* [Fix] Core: Fix processing of NDNs of certain type
* [Fix] Core: Implement logic to find some bad characters in URLs
* [Fix] Core: treat nodes with ttl properly in lru cache
* [Fix] Fix Content-Type parsing
* [Fix] Fix HTTP headers signing case
* [Fix] Fix control interface
* [Fix] Fix deletion of the duplicate headers
* [Fix] Fix emails filtering in emails module
* [Fix] Fix greylisting log message and logic
* [Fix] Fix issues with storing of the accepted addr in rspamd control
* [Fix] Fix maps object update race condition
* [Fix] Fix memor leaks and whitespace processing
* [Fix] Fix processing of null bytes in headers
* [Fix] Fix rcpt_mime and from_mime in user settings
* [Fix] Fix rfc2047 decoding for CD headers
* [Fix] Fix rfc2231 for Content-Disposition header
* [Fix] Fix setting of the subject pattern in config
* [Fix] Greylist: fix records checking
* [Fix] HTML: Another HTML comments exception fix
* [Fix] HTML: Another entities decoding logic fix
* [Fix] HTML: Fix HTML comments with many dashes
* [Fix] HTML: Fix entities in HTML attributes
* [Fix] HTML: Fix some more SGML tags issues
* [Fix] Ignore whitespaces at the end of value in DKIM records
* [Fix] MID module: Fix DKIM domain matching
* [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config
* [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge
* [Fix] Mime_parser: Fix parsing of mime parts without closing boundary
* [Fix] Multimap: Fix operating with userdata
* [Fix] Process orphaned `symbols` section
* [Fix] Rdns: Fix multiple replies in fake replies
* [Fix] Rework groups scores definitions
* [Fix] Set proper element when reading data from Sentinel
* [Fix] Set rspamd user to initialise supplementary groups on reload
* [Fix] Settings: Fix selectors usage
* [Fix] Sort data received from Sentinel to avoid constant replacing
* [Fix] groups.conf - filename typo
* [Fix] lua_scanner - oletools typos, logging
* [Fix] lua_scanners - actions and symbol_fail
* [Fix] lua_scanners - fix luacheck
* [Fix] lua_scanners - kaspersky - response with fname
* [Fix] lua_scanners - savapi redis prefix
* [Fix] tests - antivirus - fprot symbols
* [Project] Add concept of flexible actions
* [Project] Add heuristical from parser to received parser
* [Project] Add new flags to clickhouse, redis and elastic exporters
* [Project] Attach new received parser
* [Project] Fallback to callbacks from coroutines
* [Project] Implement keep-alive support in lua_http
* [Project] Lua_udp: Implement fully functional client
* [Project] Plug keepalive knobs into http connection handling
* [Project] Rspamadm: Add `modify` tool
* [Rework] Convert rspamd-server to a shared library
* [Rework] Dcc: Rework DCC plugin
* [Rework] Enable explicit coroutines symbols
* [Rework] Rework telephone urls parsing logic
* [Rework] Rewrite RBL module
* [Rework] Settings: Rework settings check
* [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir
* [Rework] Unify task_timeout
* [Rework] Use VEX instructions in assembly, relocate
* [WebUI] Notify user if uploaded data was not learned
* [WebUI] Remove redundant condition
1.8.3: 03 Dec 2018
* [CritFix] Make flags mutually exclusive for mime parts
* [CritFix] Strictly deny unencoded bad utf8 sequences in headers
* [Feature] Add Kaspersky antivirus support
* [Feature] Add method to get dkim results
* [Feature] Add more words regexp classes
* [Feature] Allow to choose words format in `rspamadm mime`
* [Feature] Allow to get all types of words from Lua
* [Feature] Allow to get task flags in C expressions
* [Feature] Allow to require encryption when accepting connections
* [Feature] Ignore bogus whitespaces in the words
* [Feature] Implement more strict configuration tests
* [Feature] Improve SPF results in Authentication-Results
* [Feature] Support ClickHouse database
* [Fix] Add failsafety for utf8 regexps
* [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts
* [Fix] Emit error if connection has been terminated with no stop pattern
* [Fix] Fix boundaries checks in embedded messages
* [Fix] Fix double free
* [Fix] Perform policy downgrade on sample out, add tests
* [Fix] Properly escape utf8 regexps in hyperscan mode
* [Fix] Selectors - attachments args condition
* [Fix] Some fixes for raw parts
* [Fix] Treat learning errors as non-fatal
* [Fix] Use tld when looking for DKIM domains
* [Project] Words unicode structure rework
* [Project] Add preliminary Redis Sentinel support
* [Project] Improve Authentication-Results header
* [Project] Rework DKIM checks results
* [Project] Use more generalised API to produce meta words
1.8.2: 19 Nov 2018
* [Conf] Add DWL support in the default configuration
* [Conf] Disable rspamd_update by default (again)
* [Conf] Fix configuration sample for ratelimit
* [CritFix] Disable broken url tags by default
* [CritFix] Fix \0 processing when doing RSA sign
* [CritFix] Fix adding symbols to their primary groups
* [Feature] Add `rspamadm cookie` utility
* [Feature] Add specialised functions for generating encrypted cookies
* [Feature] Add support of cookies in replies module
* [Feature] Add support of words regexps
* [Feature] Allow to add 3rd party clang plugins
* [Feature] Allow to create lua regexps from glob or plain patterns
* [Feature] Allow to set custom limits for upstream lists
* [Feature] Detect orphaned parts and attach them to message
* [Feature] Filter tokens in bayes
* [Feature] Fold b= value when doing arc sealing
* [Feature] Ignore cookies in the future and too old in the past
* [Feature] Skip stop words in statistics
* [Feature] Store stop words and allow to query them
* [Feature] Support query arguments in controller's custom commands
* [Feature] Tune upstream limits in Rspamd proxy
* [Feature] Use different callback symbols for different uribls
* [Feature] Write DKIM selector in dkim allow/reject symbols
* [Fix] Add obs_fws state support to eoh state machine
* [Fix] Add sanity check when applying mime boundaries heuristic
* [Fix] Antivirus - virus names with 0 were recognized as tables
* [Fix] Disable headernames in bayes temporarily
* [Fix] Do not allow syntax errors in include files...
* [Fix] Do not allow to merge an object with an array (or vice versa)
* [Fix] Don't perform forged recipients check for missing recipients
* [Fix] Fix DKIM based RBLs
* [Fix] Fix actrie implementation (sync from upstream), fixed OOB read
* [Fix] Fix explicit methods call in selectors
* [Fix] Fix extraction of additional parts
* [Fix] Fix finalization for internal plugins
* [Fix] Fix override_defaults function
* [Fix] Fix squeezed symbols when using settings
* [Fix] Fix urls insertion in Clickhouse module
* [Fix] Furhter fixes to ratelimits logic
* [Fix] Ignore signatures when looking for boundaries
* [Fix] Properly set learned count
* [Fix] Really fix ratelimits configuration and work
* [Fix] Remove ambigious format flag from printf
* [Fix] Restore URLs exporting in ClickHouse plugin
* [Fix] Rework bayes calculations...
* [Fix] Switch from chi-square to naive for large Fisher value
* [Fix] Treat normal password as enable password if there is no enable password
* [Fix] Use proper syntax for making DNS requests
* [Fix] Various fixes in embedded plugins
* [Project] Change fuzzy check selection logic to lua_fuzzy library
* [Project] Rework async events and symbols
* [Project] Move all metatokens in Bayes to lua_stat from C
* [WebUI] Add history rows per page control
1.8.1: 16 Oct 2018
* [CritFix] Fix options insertion
* [CritFix] Fix words decay one more time (affects long messages)
* [CritFix] Increase default words_decay
* [CritFix] Plug memory leak in redis pool
* [Feature] Add `check_violation` feature to DKIM/ARC signing
* [Feature] Add only unique elements to Clickhouse url arrays
* [Feature] Allow `g+:` and `g-:` composite atoms
* [Feature] Allow dkim domains check in surbl
* [Feature] Allow maps with HTTP auth
* [Feature] Allow to disable actions by users settings
* [Feature] Extend whitelisting options
* [Feature] Store url object in images
* [Feature] Use verdict instead of the plain action in plugins
* [Fix] Allow to call fstring append with NULL string
* [Fix] DCC - luacheck
* [Fix] Do not load torch on each rspamadm invocation
* [Fix] Fix boundaries detection and rework stop words algorithm
* [Fix] Fix dependencies for DNS_SIGNED symbol
* [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit
* [Fix] Fix groups mess
* [Fix] Fix groups mess
* [Fix] Fix parsing address with comments
* [Fix] Fix resolving in DMARC reports
* [Fix] Fix various issues with parsing of the received headers
* [Fix] Fix watchers issue in lua_tcp when doing no resolving
* [Fix] Plug memory leak in language detector (affects reloads)
* [Fix] Remove one letter stop words
* [Fix] Slashing: backport chunk logic from libucl
* [Fix] Stop libevent from using cached time in rspamadm
* [Fix] Try to fix watchers chaining
* [Fix] Various fixes in redis sync interface
* [Fix] ip_score - respect check_authed and check_local settings from config
* [Project] Rework passthrough actions
* [Project] Clustering module
* [Rework] Always create result for a task
* [Rework] Completely rewrite DMARC checks logic
* [Rework] Rework and fix whitelist plugin
* [WebUI] Add symbols sorting buttons
* [WebUI] Change symbols order without updating history
* [WebUI] Colorize symbols
* [WebUI] Do not display password form when secure_ip is set
* [WebUI] Fix symbol description tooltips display
* [WebUI] History: add sorting by symbol score value
- [Feature] Add arguments schemas to processors and extractors
- [Feature] Add functional selectors library
- [Feature] Add generic selector to reputation module
- [Feature] Add more ratelimits: by digest, by attachments data, by
filenames
- [Feature] Add preliminary stop words detection support
- [Feature] Add pure Lua debugm function
- [Feature] Add schema validation for Redis settings
- [Feature] Add selectors combine function
- [Feature] Add some recursion protection to lua logger
- [Feature] Add support for Lua API tracing
- [Feature] Allow to apply schema to arguments
- [Feature] Allow to get dkim signing data directly from HTTP headers
- [Feature] Allow to reuse existing authentication results
- [Feature] Cache selectors results in re runtime
- [Feature] Implement new text tokenizer based on libicu
- [Feature] Integrate selectors framework to multimap
- [Feature] Relax FORGED_RECIPIENTS
- [Feature] Support (almost) all html entities
- [Feature] Support adding and deletion of recipients in the milter
block
- [Feature] Support gathering HTTP body from fragments in lua_http
- [Feature] Support multi flag in regexp and glob maps
- [Feature] Support selectors in ratelimit module
- [Feature] Support selectors in settings
- [Feature] Use khash in HTML parser
- [Feature] Use pure Lua debugm function
- [Fix] Add fail-safety for destroying sessions
- [Fix] Allow to add result-less fake DNS records
- [Fix] Another try to fix race conditions on config unload
- [Fix] Call Lua callback on DNS timeouts
- [Fix] Deprecate task:inc_dns_req as it is redundant
- [Fix] Do not allow events deletions on cleanup
- [Fix] Do not try to process skipped messages
- [Fix] Fix HTTP requests with no body
- [Fix] Fix another cleanup race condition
- [Fix] Fix bug in processing of pcre regexps
- [Fix] Fix byte array allocation in the pool
- [Fix] Fix crashes on task cleanup
- [Fix] Fix dynamic buckets in ratelimits
- [Fix] Fix endless loop when waiting for Rspamd to stop
- [Fix] Fix lua_util.str_split in case of delimiters set
- [Fix] Fix more issues with watching of async events
- [Fix] Fix stop words detection and loading logic
- [Fix] Fix various corner cases for language detection
- [Fix] Fix watchers in lua_tcp
- [Fix] Fix words decay algorithm
- [Fix] Implement watchers replacement to handle nested calls
- [Fix] Save faked code into fake dns record
- [Fix] Show the proper frame when using lua_util.debugm
- [Fix] Use fake dns records in tests
- [Fix] Use unicode replacements for HTML entities
- [Fix] fixed "cannot find dependency on symbol 1" issue when using
replaced symbols in spamassassin rules
- [Fix] partition_id is not available in old versions of CH
- [Project] Add implicit conversion logic to selectors
- [Project] Add initial support for selectors in regexps
- [Project] Add method concept
- [Project] Further changes in unicode operations
- [Project] Implement Clickhouse migrations
- [Project] Implement implicit conversions to userdata
- [Project] Implement insert method
- [Project] Implement selectors registration for regular expressions
- [Project] Implement selectors support in re_cache
- [Project] Improve language detector: cleanup unused files,
categorize
- [Project] Migrate CH data to a fat table
- [Project] Rework selectors logic
- [Project] Start Clickhouse utilities library
- [Project] Start unicode rework
- [Project] coroutine threaded model for API calls: thread pool
- [Rework] Move phishtank to a DNS based service
- [Rework] Rework Clickhouse plugin to use the new API
- [Rework] Rework language detector
- [Rework] Rework utf content processing in text parts
- [WebUI] Add progress bar for AJAX requests
- [WebUI] Avoid errors table reinitialization
- [WebUI] Avoid history table reinitialization
- [WebUI] Avoid throughput summary table reinitialization
- [WebUI] Destroy summary table on disconnect
- [WebUI] Fix "auth" request URL
- [WebUI] Fix disabling and hiding controls on page reload
- [WebUI] Fix maps loading from neighbours
- [WebUI] Fix symbols sorting by score
- [WebUI] Fix tables destroying
- [WebUI] Fix throughput data consolidation
- [WebUI] Fix upload buttons disabling
- Fix missing config files (pkg/53577).
The most important features and fixes
- Ratelimits are reworked and now work as intended (and documented)
- Clickhouse module supports data retention policies
- Reworked C modules to avoid global contexts (simplifies leaks
detection on reload)
- Reputation plugin now supports SPF records reputation
- WebUI code is now even more conformant to the modern JS standards
- Maps are now distributed remotely with local file safety fallback to
allow faster maps update without waiting for a new release
- Antivirus module checks attachments only (as decoded content) in
attachments_only mode to improve AV performance by hiding the mime
content from them
Full list of the meaningful changes
- [CritFix] Fix caseless comparison of equal length strings
- [Feature] Add HTTP basic auth support to elastic and clickhouse
plugins
- [Feature] Add SPF selector to reputation
- [Feature] Add support of the fallback backends for HTTP maps
- [Feature] Allow to print full mime structure when extracting mime
data
- [Feature] Allow to split symbols in reputation plugin
- [Feature] Check attachments only on AV scanners in attachments_only
mode
- [Feature] Disable all SSL checks if ssl_no_verify flag is set
- [Feature] Implement parsing of scoped IPv6 addresses
- [Feature] Improve rspamc counters output
- [Fix] Add sanity checks when expanding SPF macros
- [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack)
- [Fix] Avoid one extra byte writing
- [Fix] Deal with direct hash table
- [Fix] Detect empty text part as text, not HTML
- [Fix] Do not reduce map watch timeout for mixed http/file maps
- [Fix] Fix HTML part detection heuristic
- [Fix] Fix double free in redirectors cleanup
- [Fix] Fix legacy history handling in the controller
- [Fix] Fix messages insertion
- [Fix] Fix sending string method
- [Fix] Fix statconver command line arguments
- [Fix] Fixed argument checking for being null
- [Fix] Fixed issues reported by luacheck
- [Fix] Freeze updates queue when do actual storage update
- [Fix] HTTP map hash is per-backend and not per-map
- [Fix] Plug memory leak in fuzzy updates
- [Fix] Prefer 'MTA-Name' when producing authentication results
- [Fix] Replace bad unicode sequences instead of stopping on them
- [Fix] Set classifier version on learning
- [Project] Reworked ratelimits
- [Project] Apply topological sorting for symbols in Rspamd
- [Project] Remove global contexts from C modules
- [Project] Move performance critical hash tables to khash
- [WebUI] Avoid unused indexes
- [WebUI] Do not execute on_success callback
- [WebUI] Fix history reset for "All SERVERS" (#2346)
- [WebUI] Fix query URL for selected server
- [WebUI] Fix symbols display in legacy history,
- [WebUI] Hide symbols order selector for legacy history
- [WebUI] Refactor query functions into one
- [WebUI] Remove previously-attached event handlers
- [WebUI] Save symbols to the selected server
- [WebUI] Unify arguments of query functions
- [WebUI] Use common query functions to get graph data
- [WebUI] Use common query functions to save symbols
1.7.8: 12 Jul 2018
- [Feature] Add more extended statistics about fuzzy updates
- [Feature] Add more non-conformant Received headers support
- [Feature] Add preliminary function to get fuzzy hashes from text in
Lua
- [Feature] Allow to configure AV module rejection message
- [Feature] Implement fuzzy hashes extraction in mime tool
- [Feature] Improve WHITE_ON_WHITE rule
- [Feature] Improve integer -> string conversion
- [Feature] Reuse maps in multimap module more aggressively
- [Fix] Avoid race condition in skip map as pool lifetime is not
enough
- [Fix] Eliminate all specific C plugins pools
- [Fix] Fix DKIM check rule if DNS is unavailable
- [Fix] Fix build where ucontext is defined in ucontext.h
- [Fix] Fix crash in base url handling
- [Fix] Fix descriptors leak in sqlite3 locking code
- [Fix] Fix messages quarantine
- [Fix] Fix padded numbers printing
- [Fix] Fix race condition on maps reinit
- [Fix] Fix regexp functions when no data is passed
- [Fix] Fix specific urls extraction
- [Fix] Fix styles propagation
- [Fix] Improve resetting of the limit buckets
- [Fix] Initialize sqlite3 properly
- [Fix] Work with broken resolvers in resolv.conf
- [Project] Implement HTTP maps caching
- [Project] Refresh fuzzy hashes when matched
- [Project] Add logic to deduplicate fuzzy updates queue
- [WebUI] Add missed declarations
- [WebUI] Avoid using "undefined" property
- [WebUI] Do not accept passwords containing control characters
- [WebUI] Do not redeclare variables
- [WebUI] Enable strict mode,
- [WebUI] Fix variable assignment
- [WebUI] Initialize variables at declaration
- [WebUI] Remove duplicated path from RequireJS config
- [WebUI] Remove unused block
- [WebUI] Remove unused variable
- [WebUI] Remove unused variables
- [WebUI] Use self-explanatory notation
- [WebUI] Use type-safe equality operators
1.7.7: 02 Jul 2018
- [CritFix] Check NM part of pubkey to match it with rotating keypairs
- [CritFix] Do not overwrite PID of the main process
- [CritFix] Fix maps after reload
- [CritFix] Fix maps race conditions on reload
- [CritFix] Fix shmem leak in encrypting proxy mode
- [Feature] Add a concept of ignored symbols to avoid race conditions
- [Feature] Add ability to print bayes tokens in rspamadm mime
- [Feature] Add method to get statistical tokens in Lua API
- [Feature] Add preliminary mime stat command
- [Feature] Add rspamadm mime tool
- [Feature] Add urls extraction tool
- [Feature] Address ZeroFont exploit
- [Feature] Allow rspamadm mime to process multiple files
- [Feature] Allow to extract words in `rspamadm mime`
- [Feature] Allow to print mime part data
- [Feature] Allow to show HTML structure on extraction
- [Feature] Distinguish IP failures from connection failures
- [Feature] Improve output for mime command
- [Feature] Improve styles propagation
- [Feature] Main process crash will now cleanup all children
- [Feature] Preload file and static maps in main process
- [Feature] Print stack trace on crash
- [Feature] Process font size in HTML parser
- [Feature] Propagate content length of invisible tags
- [Feature] Read ordinary file maps in chunks to be more safe on
rewrites
- [Feature] Support base tag in HTML
- [Feature] Support more size suffixes when parsing HTML styles
- [Feature] Support opacity style
- [Fix] Another fix for nested composites
- [Fix] Fill nm id in keypairs cache code
- [Fix] Fix colors alpha channel handling
- [Fix] Fix destruction logic
- [Fix] Fix double free
- [Fix] Fix maps preload logic
- [Fix] Fix nested composites process
- [Fix] Fix proxying of Exim connections
- [Fix] Fix reload crash
- [Fix] Fix rspamadm -l command
- [Fix] Update ed25519 signing schema
- [WebUI] Stop using "const" declaration
- [WebUI] Update RequireJS to 2.3.5
1.7.6: 15 Jun 2018
- [CritFix] Fix multiple neural networks support
- [Feature] Add decryption function to keypair command
- [Feature] Add gzip compression for HTTP requests in elastic module
- [Feature] Add gzip methods to lua util
- [Feature] Add maps based on Top Level Domains
- [Feature] Add pubkey checks for dkim_signing
- [Feature] Add support of fake DNS records
- [Feature] Add tool to encrypt files
- [Feature] Allow to add symbols using settings directly
- [Feature] Allow to match private and public keys for DKIM signatures
- [Feature] Allow to set task flags via settings
- [Feature] Allow to specify fake DNS address from the config
- [Feature] Implement signatures verification using rspamadm keypair
- [Feature] Implement signing using `rspamadm keypair`
- [Feature] Improve error reporting for DKIM key access issues
- [Feature] Provide $HOSTNAME variable in UCL
- [Feature] Rework levenshtein distance computation
- [Feature] Split message parsing and processing
- [Feature] Support ED25519 DKIM signatures
- [Feature] Support encrypted configs in UCL
- [Feature] Suppress duplicate warning on very large radix tries
- [Feature] Use OSB to combine header names
- [Fix] Cleanup maps data on shutdown
- [Fix] Fix '~' behaviour in composites
- [Fix] Fix HTTP maps updates
- [Fix] Fix NIST signatures
- [Fix] Fix RFC822 comments when processing a mime address
- [Fix] Fix double free
- [Fix] Fix dynamic settings application
- [Fix] Fix for CommuniGate Pro maillist
- [Fix] Fix keypair creation method to actually create keypair...
- [Fix] Fix matching patterns with no paths
- [Fix] Fix memory leak in parsing comments
- [Fix] Fix parsing of urls with numeric password
- [Fix] Fix plugins intialisation in configwizard
- [Fix] Fix potential crash on reload
- [Fix] Fix potential race condition for a finished HTTP connections
- [Fix] Fix race-condition leak on processes reload
- [Fix] Fix signing in openssl mode
- [Fix] Free language detector structures
- [Fix] Relax alignment requirements
- [Fix] Send DMARC reports compressed
- [Fix] Try to fix leak in dmarc module
- [Fix] Try to plug memory leak in metric exporter
- [Project] Convert rspamadm subcommands to Lua
- [WebUI] Display smtp sender/recipient in history
- [WebUI] Fix elements disabling in "Symbols" tab
- [WebUI] Limit recipients list in history column to 3
- [WebUI] Match envelope and mime addresses following in arbitrary
order
- [WebUI] Update column header
- [WebUI] Wrap addresses in history
1.7.5: 18 May 2018
- [Conf] Add MSBL proposed return codes
- [Conf] Add additional groups for policies
- [CritFix] Do not use volatile Lua strings as UCL keys
- [Feature] Add ability to add fuzzy hashes to headers
- [Feature] Add function to extract most meaningful urls
- [Feature] Add rule to block mixed text and encrypted parts
- [Feature] Allow multiple groups for symbols
- [Feature] Allow to disable lua squeezing logic
- [Feature] Allow to get multipart children in Lua
- [Feature] Allow to insert multiple headers from milter headers
- [Feature] Allow to print scores in subject and further extensions
- [Feature] Be more error-prone in squeezed rules
- [Feature] Support multiple return codes in emails module
- [Feature] Use EMA for calculating averages
- [Feature] Use common jit cache for all regexps
- [Feature] support for CommuniGate Pro self-generated messages
- [Fix] Allow to have multiple values for headers as arrays
- [Fix] Do not open sockets for disabled workers
- [Fix] Fix AuthservId
- [Fix] Fix base64 folding in Lua API
- [Fix] Fix build on non-x86 platforms
- [Fix] Fix cached maps logic
- [Fix] Fix compatibility with old maps query logic
- [Fix] Fix crash if skip_map is used
- [Fix] Fix importing static maps from UCL
- [Fix] Fix parsing of unix sockets
- [Fix] Fix raw_mime regexp on HTML part with no text content
- [Fix] Fix tables logging
- [Fix] Fix vertical tab handling in libucl
- [Fix] Try to fix frequency counters
- [Fix] Use better sharding for ip_score
- [Fix] Use multiple results from SURBL DNS reply
- [Fix] When doing AV scan select a different server for retransmit
* [Conf] Major stock config updates:
- Workers are now specified in a new format worker "type" { ... }
- Enable fuzzy worker to simplify local fuzzy storages configuration
- Bind all workers to localhost by default to avoid security flaws
* [Conf] Make more sane fuzzy_check default settings
* [CritFix] Fix ucl escape for bad symbols
* [Feature] Add failure symbol for AV module
* [Feature] Add lazy expiration mode for new classifier schema
* [Feature] Add preliminary version of maps stats plugin
* [Feature] Allow to block fuzzy requests from specific networks
* [Feature] Allow to change `expire` of live statistics
* [Feature] Distinguish AV failure from clean result
* [Feature] Further improvements of language detector by using khash
* [Feature] Further optimization of the lang_detection
* [Feature] Implement cluster-aware bayes expiry
* [Feature] Implement exclude patterns in rspamc
* [Feature] Implement glob maps in addition to regexp maps
* [Feature] Implement map statistics function for lua API
* [Feature] Implement stop symbols for Clickhouse collection
* [Feature] Support recipients separated by commas
* [Feature] Try harder to upload scripts to the Redis server
* [Feature] Upgrade t1ha distribution
* [Feature] use_domain_sign_inbound
* [Feature] Use scores from maps if `symbols_set` is not defined
* [Fix] Add resolving version of radix map helper
* [Fix] Check URL before adding implicit prefix
* [Fix] Do not check pid/state when using PRNG
* [Fix] Fix CentOS logrotate script for systemd
* [Fix] Fix slash + dot in urls
* [Fix] Fix systemd version of the logrotate script
* [Fix] Propagate key when import implicit array from Lua
* [Fix] Strip spaces from map keys and values
* [Fix] Try to fix a specific case when processing milter protocol
* [Fix] Try to fix crash when a tcp connection cannot be set
* [Fix] Typo use_domain_local --> use_domain_sign_local
* [Fix] Various fixes to once_received module
* [Project] Store hits counters for map elements
- [CritFix] Plug bad memory leak in protocol reply
- [Feature] Add avx2 codec for base64
- [Feature] Add method to receive all URL flags from Lua API
- [Feature] Allow to fold headers on stop characters
- [Feature] Allow to set lua_cpath from options
- [Feature] Allow to specify custom rejection message in milter
- [Feature] Deal with unnormalised Unicode obfuscation
- [Feature] Do not detect language twice for relative parts
- [Feature] Implement oversigning feature
- [Feature] Implement silent logging level to minimize noise in logs
- [Feature] Improve URL_IN_SUBJECT rule
- [Feature] Use hashing to reduce redis attack surface
- [Fix] Add oversigning for the most important headers
- [Fix] add 'rewrite subject' to History dropdown
- [Fix] Another fix in folding algorithm
- [Fix] Do not call multimap addr for parts of addr if filter is
presented
- [Fix] Do not clean hostname on generic reset
- [Fix] Do not create pid file in no-fork mode
- [Fix] Fix fold_after case to preserve multiple spaces
- [Fix] Fix folding and folding tests
- [Fix] Fix hostname usage in milter mode
- [Fix] Fix lua RSA verify and its tests
- [Fix] Fix metadata exporter send_mail backend (#2124)
- [Fix] Fix processing of '\v' in libucl
- [Fix] Fix shemaless URLs detection
- [Fix] Fix support of multiple headers in sign_header
- [Fix] Fix usage of util.parse_mail_address
- [Fix] Fix weights of dynamic squeezed rules
- [Fix] Leak from bucket before checking the burst
- [Fix] Stop using own localtime as DST could be messy in many cases
- [Fix] Treat unnormalised URLs as obscured
- [Rework] Restore leaky bucket model in ratelimit plugin
- [WebUI] Add messages total to throughput summary
- [WebUI] Add symbols order selector to history
- [WebUI] Config: Load list on demand
- [WebUI] Fix modalBody for maps that appear more than once
- [WebUI] History: Fix Tooltips on paging, filtering and sorting
- [WebUI] Remove a previously-attached event handler
- [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1
- [Feature] Store emails in Clickhouse
- [Feature] Support single quotes in config
- [Feature] Use templates when publishing CH schema
- [Feature] Improve Docker image
- [Fix] Add rounding when printing a lot of FP variables
- [Fix] Allow to disable certain actions by assigning null to them
- [Fix] Disable results caching
- [Fix] Fix disabling of squeezed symbols
- [Fix] Fix scan time set
- [Fix] Rework logic of actions setting
- [Fix] Try to fix various Lua stack issues
- [WebUI] Add link tag for favicon.ico
- [WebUI] Display hostname:port/path in the page title
- [CritFix] Fix lowercase comparison
- [CritFix] Timezone defines seconds WEST UTC not East
- [Feature] Add filename to log format
- [Feature] Add lua rules squeezing
- [Feature] Add related symbols analysis to rspamd_stats
- [Feature] Remove upstream `X-Spam: Yes` header by default
- [Feature] rspamd_stats: Output progress info on STDERR
- [Feature] Whitelist for emails module
- [Fix] Do not allow dependencies on self
- [Fix] Do not cache metric result
- [Fix] Do not trust all issuers as a client certificate
- [Fix] Fix dependencies in lua squeeze
- [Fix] Fix enabling/disabling squeezed rules
- [Fix] Fix enabling/disabling symbols
- [Fix] Fix external dependencies
- [Fix] Fix processing of a single compressed file
- [Fix] Fix some typos
- [Fix] Fix various modules in case of empty message
- [Fix] Handle callbacks that returns table of options
- [Fix] Improve cached action interaction
- [Fix] Make dynamic conf more NaN aware
- [Fix] Never hide actions from WebUI `configuration` tab
- [Project] Implementation of Lua rules squeezing
- [Conf] Add bayes_expiry as explicit module
- [Conf] Adjust names and weights for neural network plugin
- [Conf] Change updates url
- [Conf] Default statistics is stored in Redis now
- [Conf] Disable fann_redis module by default
- [Conf] Fix default elastic configuration
- [Conf] Fix double quote position
- [Conf] Massive config rework for new structure of symbols and scores
- [Conf] Rename Rambler BLs as they are now Rspamd's ones
- [Conf] Use dedicated rspamd.com subdomains
- [Conf] Use more data from rspamd.com fuzzy storage
- [CritFix] Add sanity guards for badly broken HTML
- [CritFix] Another errors path handling fix
- [CritFix] Another portion of tokenization fixes
- [CritFix] Do not send reject messages after set reply
- [CritFix] Fix ARC chain verification
- [CritFix] Fix crash in milter errors handler
- [CritFix] Fix memory leak in spf caching logic
- [CritFix] Fix milter commands pipelining
- [CritFix] Fix newlines detection
- [CritFix] Fix semicolons parsing in the content type
- [CritFix] Plug memory leak in zstd protocol compression
- [Feature] Add ability to match score in force_actions module
- [Feature] Add aes-rng PRF to libottery
- [Feature] Add 'composites' debug module
- [Feature] Add concept of experimental modules
- [Feature] Add DKIM trace symbol
- [Feature] Add EBL to the default config
- [Feature] Add expected ip check for emails plugin
- [Feature] Add framework to manage Redis scripts
- [Feature] Add framing for the new reputation generic plugin
- [Feature] Add function to show plugins stat
- [Feature] Add gzip compression support for clickhouse module
- [Feature] Add gzip compression support for rspamd controller
- [Feature] Add gzip support when sending lua http requests
- [Feature] Add json output for rspamd_stats
- [Feature] Add method to do a synchronous Redis connection
- [Feature] Add method to get all content-type attributes in Lua
- [Feature] Add `-m` flag to configdump to show modules states
- [Feature] Add mime types to extensions map
- [Feature] Add more features to rescore utility
- [Feature] Add more gtube like patterns to test other spam actions
- [Feature] Add more metafunctions, improve logging
- [Feature] Add more text attributes
- [Feature] Add new configwizard command to rspamadm
- [Feature] Add new tooling for stats conversation
- [Feature] Add old groups migration tool
- [Feature] Add plugins state variable
- [Feature] Add preliminary ecdsa keys support in DKIM
- [Feature] Add preliminary support of idempotent symbols
- [Feature] Add Redis server wizard
- [Feature] Add routine to convert old style stats to a new one
- [Feature] Add some sanity checks for actions and controller
- [Feature] Add statistic convertation module to configwizard
- [Feature] Add sugestions logic to mempool allocator
- [Feature] Add support of config transform in Lua
- [Feature] Add timeout to rspamc when doing corpus test
- [Feature] Add tooling to convert bayes schemas
- [Feature] Add torch conditional to configuration
- [Feature] Add torch-decisiontree package
- [Feature] Add torch-optim contrib package
- [Feature] Add TTL autodetection
- [Feature] Add urls reputation to the reputation framework
- [Feature] Allow floating and negative values in expressions limits
- [Feature] Allow multiple CTs in full extensions map
- [Feature] Allow multiple fann rules
- [Feature] Allow randomly select User-Agent from a list
- [Feature] Allow rspamadm commands to export methods in Lua
- [Feature] Allow rule specific min_bytes in fuzzy check
- [Feature] Allow to adjust symbols scores from Lua
- [Feature] Allow to attach stat signature to messages
- [Feature] Allow to change SMTP from via milter headers
- [Feature] Allow to configure monitored
- [Feature] Allow to create directories in Lua API
- [Feature] Allow to disable torch and skip train samples for ANN
- [Feature] Allow to discard messages dynamically
- [Feature] Allow to enable/disable languages from the detector
- [Feature] Allow to generate DKIM keys from rspamadm API
- [Feature] Allow to get CPU flags from Lua
- [Feature] Allow to have high precision timestamps in logs
- [Feature] Allow to insert headers into specific position
- [Feature] Allow to limit redirector requests per task
- [Feature] Allow to load and use dynamic ANNs with torch
- [Feature] Allow to quarantine rejected messages using milter
interface
- [Feature] Allow to receive signing keys from mempool vars
- [Feature] Allow to reserve elements in libucl
- [Feature] Allow to reuse signal handlers chains
- [Feature] Allow to set custom mempool variables from settings
- [Feature] Allow to set headers from settings
- [Feature] Allow to set Settings-Id for all connections
- [Feature] Allow to skip real action and add a header instead
- [Feature] Allow to skip specific hashes in fuzzy storage
- [Feature] Allow to spawn asynchronous processes from Lua
- [Feature] Allow to specify number of threads for ANN learning
- [Feature] Allow to use global lua maps in settings
- [Feature] Allow to use postfilters in composites
- [Feature] Allow to verify signatures from HTTP headers in maps
- [Feature] Antivirus: ordered pattern matches
- [Feature] Authentication-Results: support hiding usernames
- [Feature] Automatically create tables in clickhouse
- [Feature] Catch next-to-last bad extension
- [Feature] Check cached maps more frequently
- [Feature] Check groups sanity
- [Feature] Deal with obscured URLs with @ symbols
- [Feature] Enhance task:store_in_file method
- [Feature] Export password encryption routines to Redis
- [Feature] Filter nan and inf when adding scores
- [Feature] Finalize 7zip files support
- [Feature] Further improvements in language detection
- [Feature] Further improvements in language detection algorithm
- [Feature] Generic key name expansion for Redis keys
- [Feature] Hash whitelist for fuzzy_check
- [Feature] Implement bayes signatures storage
- [Feature] Implement buckets for Redis backend
- [Feature] Implement DKIM reputation adjustments
- [Feature] Implement forked workers children monitoring
- [Feature] Implement headers flags in mime parser
- [Feature] Implement l1/l2 regularization against the current weights
- [Feature] Implement manual ANN train mode
- [Feature] Implement per-user ANN support
- [Feature] Implement torch based ANN learning
- [Feature] Implement upstreams logic for clickhouse exporter
- [Feature] Import torch to Rspamd...
- [Feature] Improve allocation policy when interacting with Lua
- [Feature] Improve Lua/C interaction in history_redis
- [Feature] Improve multiple fuzzy results combining
- [Feature] Improve parsing of DKIM keys: parse algorithm
- [Feature] Improve subprocesses termination handle
- [Feature] Improve symbol type parsing in Lua API
- [Feature] Metadata Exporter: e-Mail Alerts: support multiple
recipients; alerting senders/recipients/users
- [Feature] Milter headers: support adding/removing arbitrary headers
from config
- [Feature] More metatokens
- [Feature] Multimap: checking of symbol options
- [Feature] Multimap: template URL filter
- [Feature] New bayes expiry plugin
- [Feature] Periodically save rspamd stats to disk
- [Feature] Preliminary import of the elasticsearch module
- [Feature] Ratelimit: allow full addresses in whitelisted_rcpts
- [Feature] Ratelimit: support fetching limits from Redis
- [Feature] RBL: received: filtering by position & flags
- [Feature] Read global maps for lua
- [Feature] Redis settings: support checking multiple keys
- [Feature] Rework fann plugin to be a normal post-filter
- [Feature] Rework logging configuration for rspamadm case
- [Feature] Rework short hashes generation to avoid FP
- [Feature] Save real ucl types when exporting to Lua
- [Feature] Set TCP_NODELAY for milter sockets
- [Feature] Setup DKIM signing from configwizard
- [Feature] Skip certain symbols from ANN classify
- [Feature] Store plugins state
- [Feature] Support etag for HTTP maps
- [Feature] Support Expires header when using HTTP maps
- [Feature] Support sending given header multiple times in lua_http
- [Feature] Support sha512 in DKIM signatures
- [Feature] Try to detect HTML messages better
- [Feature] Use array instead of queue to reduce memory fragmentation
- [Feature] Use controller port by default when connecting to local IP
- [Feature] Use rdtsc where possible
- [Fix] Actively load skip hashes map in fuzzy storage
- [Fix] Add another workaround to display history properly
- [Fix] Add definition for old glib compatibility method
- [Fix] Add missing rspamadm control options to help
- [Fix] Add workaround for IPv6 in sendmail
- [Fix] Add workaround for system with non-XSI compatible tzset
- [Fix] Allow oversigning in DKIM signatures
- [Fix] Allow to check negative scores in force_actions
- [Fix] Allow to have negative actions limits
- [Fix] Allow to set any layers number for fann rules
- [Fix] Another fix for rdtcs
- [Fix] Another fix to lua xmlrpc
- [Fix] Another try to deal with #1998
- [Fix] Another try to fix#1998
- [Fix] Another try to fix threading in torch
- [Fix] Apply language detection when adding fuzzy hashes
- [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer
- [Fix] Authentication Results: Fix SPF smtp.mail_from
- [Fix] Auth-Results: Multiple DKIM signatures
- [Fix] Avoid changing content-transfer-encoding header's value
- [Fix] Better handling of the legacy protocol
- [Fix] Check decoded headers sanity (e.g. by excluding \0)
- [Fix] Check for magic when checking for an archive
- [Fix] Cleanup mess with groups
- [Fix] Clickhouse: Insertion in the symbols table
- [Fix] Crash in URL processing
- [Fix] Deal with another case when processing exceptions
- [Fix] Deal with deeply nested messages more aggressively
- [Fix] Deal with nan and inf encoding in json/ucl
- [Fix] Deal with non-key arguments in lua_redis.exec_script
- [Fix] Deal with unknown weight
- [Fix] Deal with URLs with no slashes after protocol
- [Fix] Deal with URLs wrapped in [] in text parts
- [Fix] Deal with zero scores symbols
- [Fix] Default monitoring domain for surbl plugin
- [Fix] Delay upstream re-resolving when one upstream is defined
- [Fix] Detection of maillist optimized and fixed
- [Fix] DKIM signing: allow for auth_only to be false
- [Fix] DMARC: require report_settings for sending reports only
- [Fix] Do not allow garbadge when checking url domain
- [Fix] Do not cache SPF records with PTR elements
- [Fix] Do not constantly re-resolve failed upstreams with a single
element
- [Fix] Do not crash if no words defined
- [Fix] Do not crash on empty subtype
- [Fix] Do not expose spamtrap messages to SMTP reply
- [Fix] Do not fail rbl plugin when there are no received or emails
- [Fix] Do not ignore short words
- [Fix] Do not include idempotent/nostat symbols to checksum
- [Fix] Do not override groups when converting metrics
- [Fix] Do not override unix socket group when group comes before
owner
- [Fix] Do not skip the last character
- [Fix] Do not spawn too many workers by default
- [Fix] Do not stop monitored on dns errors
- [Fix] Do not stop parsing headers on bad IP header
- [Fix] Do not strip last character in the last word
- [Fix] Do not treat script content as text
- [Fix] Do not try to connect to non-supported addresses
- [Fix] Do not try to dereference last character
- [Fix] Do not try to sign unknown domains
- [Fix] Don't use whitelist/greylist maps as regexp, but as map
- [Fix] Erase unknown HTML entities
- [Fix] Exim Received header protocol parsing
- [Fix] First load selector_map and path_map. And only return false
when domain not found if try_fallback is false
- [Fix] Fix a lot of FP in chartable in mixed languages
- [Fix] Fix ANN checks
- [Fix] Fix ANN loading logic
- [Fix] Fix another tokenization issue
- [Fix] Fix autolearn parameters reading
- [Fix] Fix bad archive characters stripping
- [Fix] Fix bad extension check
- [Fix] Fix bayes schema conversion
- [Fix] Fix blacklists and DMARC in whitelist
- [Fix] Fix brain-damaged torch build system
- [Fix] Fix build on FreeBSD
- [Fix] Fix clickhouse exporter
- [Fix] Fix clickhouse schema
- [Fix] Fix comparision
- [Fix] Fix composites processing
- [Fix] Fix connecting to a unix socket in rspamadm statconvert
- [Fix] Fix couple of warnings
- [Fix] Fix crashes in the rspamd_control path
- [Fix] Fix deletion from hash
- [Fix] Fix DKIM forgeries via multiple headers
- [Fix] FIx dynamic conf plugin
- [Fix] Fix emails detection
- [Fix] Fix empty headers simple canonicalization
- [Fix] Fix empty threshold check in greylisting module
- [Fix] Fix encrypted legacy reply in fuzzy storage
- [Fix] Fix enormous scores for R_WHITE_ON_WHITE
- [Fix] Fix exceptions list in surbl
- [Fix] Fix *_EXCESS_BASE64 rules
- [Fix] Fix expire rounding
- [Fix] Fix extra hits in PCRE mode for regular expressions
- [Fix] Fix format strings
- [Fix] Fix get_content method
- [Fix] Fix groups override when defining symbols
- [Fix] Fix learned count in new schema
- [Fix] Fix learn errors propagation
- [Fix] Fix loading of per-user redis backend for statistics
- [Fix] Fix logging buffer corruption in case of repeated messages
- [Fix] Fix lua cached elements invalidation
- [Fix] Fix merging of the implicit arrays
- [Fix] Fix mime_types scoring
- [Fix] Fix multiple headers in DKIM headers list
- [Fix] Fix null callee case in clang plugin
- [Fix] Fix obscured url in format user@@example.com
- [Fix] Fix parsing of the per-user script
- [Fix] Fix priorities in rspamd_update, disable rules execution
- [Fix] Fix processing of closed tags
- [Fix] Fix processing of idempotent rules when autolearn fails
- [Fix] Fix processing of multipart parts with no headers
- [Fix] Fix processing of skip-hashes in fuzzy storage
- [Fix] Fix PTR processing in SPF
- [Fix] Fix pushing country to clickhouse asn table
- [Fix] Fix random forests module
- [Fix] Fix real IP parsing for some strange Exim received
- [Fix] Fix Redis timeout setup
- [Fix] Fix reload crash when hyperscan is enabled
- [Fix] Fix reusing of redis connection after exec
- [Fix] Fix sanity checks on macro value
- [Fix] Fix setting of path and cpath for Lua
- [Fix] Fix setting of signals when spawning a thread
- [Fix] Fix text splitting: stack overflow (too many captures)
- [Fix] Fix ticks processing
- [Fix] Fix upstream addrs updating
- [Fix] Fix urls/emails distinguishing found in queries
- [Fix] Fix user settings check
- [Fix] Fix variable increment
- [Fix] Fix various issues in stat_convert
- [Fix] F-PROT Antivirus infection string for all known occurences
- [Fix] F-PROT Antivirus: only check return code to determine
infection
- [Fix] Further fixes around floating point expressions
- [Fix] Further fixes to ANN module
- [Fix] Further fixes to rescore tool
- [Fix] Further fixes to support ES 6
- [Fix] Further tokenization fixes
- [Fix] Greylisting set phase is not idempotent
- [Fix] Handle proxy copy errors
- [Fix] Header checks: Fix get_raw_header method
- [Fix] Header checks: REPLYTO_UNPARSEABLE rule
- [Fix] Kill spawned processes on termination
- [Fix] Load skip map from all processes as shared cache is
unavailable
- [Fix] Lowercase HTTP headers to make them searchable from Lua
- [Fix] Lowercase words
- [Fix] Lua_http: freeing
- [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid
"rspamd_config_read: rcl parse error: cannot init lua file [...]
module 'lpeg' not found"
- [Fix] Map absence is not an error
- [Fix] Metadata exporter: check IP sanity
- [Fix] Milter headers: custom headers: removing headers
- [Fix] Milter headers: skip_local / skip_authenticated settings
- [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first
- [Fix] mime_types: fix next-to-last extension length check
- [Fix] More hacks to deal with old configs
- [Fix] Move composites second pass to the dedicated stage
- [Fix] Multimap: received: filtering of artificial header
- [Fix] Multiple fixes in torch based ANN plugins
- [Fix] Once more fix bad extension check
- [Fix] Optimize rspamd_fstring_t reallocations
- [Fix] options.local_networks setting
- [Fix] Parse HREF urls without explicit prefix
- [Fix] Plan new event on HTTP errors
- [Fix] Plug another possible memory leak
- [Fix] Plug memory leak
- [Fix] Plug memory leak in lua_tcp
- [Fix] Plug memory leak when setting email addresses from Lua
- [Fix] Propagate learn/stat errors more precisely
- [Fix] Ratelimit: fix whitelisted_rcpts matching
- [Fix] Ratelimit: lowercase email addresses
- [Fix] RBL: received: deal with missing data
- [Fix] Rebalance and slightly rework MX check plugin
- [Fix] Redis key expansion: EVAL: deal with strings
- [Fix] Redis script loading in DMARC; URL tags; URL reputation
- [Fix] Reject invalid bh for DKIM signatures earlier
- [Fix] Relax pem signature detection
- [Fix] Relax unicode properties requirements for chartable module
- [Fix] Remove extra noise from dkim and arc signing
- [Fix] Remove hop-by-hop headers in proxy
- [Fix] Remove incorrect method `task:set_metric_subject`
- [Fix] Replace space like characters in headers with plain space
- [Fix] Restore old style ratelimits support
- [Fix] Rework elasticsearch plugin
- [Fix] Rewriting subjects via force actions module
- [Fix] RPM postinstall
- [Fix] Sanitize IP in history redis
- [Fix] Select the correct signature when doing simple canon
- [Fix] Set CLOEXEC flag on files opened
- [Fix] Setting check_local / check_authed in plugins
- [Fix] Settings: avoid checking invalid IP
- [Fix] Settings: header: deal with multiple settings
- [Fix] Skip checks if both extensions are not bad
- [Fix] Skip nostat tokens when get number of tokens
- [Fix] Some more fixes towards emails detection
- [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't
match
- [Fix] Stop using of g_slice...
- [Fix] Switch rspamadm logging to message level
- [Fix] Symbol 'FANNR_SPAM' has its score defined..
- [Fix] Table parameter for rspamd_config:add_doc()
- [Fix] Treat 'rewrite subject' as spam action
- [Fix] Try harder in passing IPv6 addresses
- [Fix] Try harder to find rfc822 notifications
- [Fix] Try harder to find urls
- [Fix] Use decoded values when parsing mime addresses
- [Fix] Use full URL when making an HTTP request
- [Fix] Use greylisting threshold in greylisting module
- [Fix] Use n_words attribute from ngramms
- [Fix] Use raw urls when sending requests to redirector
- [Fix] Use the right boolean operator on error check
- [Fix] Use weight from map for fuzzy scoring
- [Fix] Various fixes to elastic plugin
- [Fix] Various fixes to fann_redis instantiation
- [Fix] Various improvements in language detection
- [Fix] Virus infection string for F-PROT Antivirus
- [Fix] Virus infetction string for F-PROT Antivirus
- [Fix] WebUI: use relative path for savemap
- [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the
rule
- [Fix] Write configuration changes as UCL config
- [Project] Add detection logic for words
- [Project] Add fast debug logging infrastructure
- [Project] Add more flags to languages
- [Project] Add n-gramms data files
- [Project] Add ngramms frequencies detector
- [Project] Add random words selection logic
- [Project] Add unigramms to language detection as well
- [Project] Convert all C modules to fast debug infrastructure
- [Project] Detect some languages based on unicode script
- [Project] Enable fast debug lookup for some modules
- [Project] Enable language detector init in scanner workers
- [Project] Further improvements to language detector
- [Project] Implement logic of ngramms application
- [Project] Improve weighting in lang_detection
- [Project] Initialize language detector
- [Project] Preliminary version of ngramms based language detector
- [Project] Preliminary version of the new stat_convert
- [Project] Remove old language detector
- [Project] Rework language detection ngramms structure
- [Project] Start language detection project
- [Project] Start rework of language detection to improve quality
- [Project] Use fast debug logging check
- [Rework] Add frame for new reputation based IP score module
- [Rework] Continue stat_convert rework task
- [Rework] Implement new version of fuzzy replies
- [Rework] Improve readability of xmlrpc API
- [Rework] Kill metrics!11
- [Rework] Ratelimit module
- [Rework] Rename fann_redis to neural plugin
- [Rework] Reorganize mime_types module
- [Rework] Rework rescore utility
- [Rework] Rewrite model and learning logic for rescore
- [Rework] Run post-loads when all initialization is completed
- [Rework] Simplify lua path initialization
- [Rework] Start major stat_convert rework
- [Rework] Start mempool fragmentation reduce project
- [Rework] Start moving of fann redis to torch
- [Rework] Stop embedding rspamadm scripts into C
- [Rework] Use floating point arithmetics in Rspamd expressions
- [Rework] Use frequencies distribution in language detector
- [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages
- [WebUI] Compact graph selectors
- [WebUI] Escape strings inside HTML in history
- [WebUI] Fix message count in throughput summary
- [WebUI] Fix NaNs display on Throughput graph
- [WebUI] Migrate widgets to D3 v4
- [WebUI] Restore passwordless login support
- [WebUI] Show symbol descriptions as tooltips in history
- [WebUI] Stop using commas in pie chart tooltips
- [WebUI] Update D3 and jQuery
- [WebUI] Update D3Evolution 1.0.0 -> 1.1.0
- [CritFix] Add sanity guards for badly broken HTML
- [CritFix] Another errors path handling fix
- [CritFix] Fix ARC chain verification
- [CritFix] Fix crash in milter errors handler
- [Feature] Allow to insert headers into specific position
- [Feature] Allow to receive signing keys from mempool vars
- [Feature] Authentication-Results: support hiding usernames
- [Fix] Another try to deal with #1998
- [Fix] Another try to fix#1998
- [Fix] Better handling of the legacy protocol
- [Fix] Check decoded headers sanity (e.g. by excluding \0)
- [Fix] Deal with nan and inf encoding in json/ucl
- [Fix] Deal with URLs wrapped in [] in text parts
- [Fix] DKIM signing: allow for auth_only to be false
- [Fix] Do not crash on empty subtype
- [Fix] Do not fail rbl plugin when there are no received or emails
- [Fix] Do not skip the last character
- [Fix] Do not try to dereference last character
- [Fix] Do not try to sign unknown domains
- [Fix] Exim Received header protocol parsing
- [Fix] First load selector_map and path_map. And only return false
when domain not found if try_fallback is false
- [Fix] Fix bad archive characters stripping
- [Fix] Fix comparision
- [Fix] Fix connecting to a unix socket in rspamadm statconvert
- [Fix] Fix empty headers simple canonicalization
- [Fix] Fix extra hits in PCRE mode for regular expressions
- [Fix] Fix parsing of the per-user script
- [Fix] Fix processing of skip-hashes in fuzzy storage
- [Fix] Fix Redis timeout setup
- [Fix] Fix sanity checks on macro value
- [Fix] Fix text splitting: stack overflow (too many captures)
- [Fix] Fix urls/emails distinguishing found in queries
- [Fix] F-PROT Antivirus: only check return code to determine
infection
- [Fix] Metadata exporter: check IP sanity
- [Fix] Multimap: received: filtering of artificial header
- [Fix] Plan new event on HTTP errors
- [Fix] Plug another possible memory leak
- [Fix] Remove hop-by-hop headers in proxy
- [Fix] Sanitize IP in history redis
- [Fix] Setting check_local / check_authed in plugins (#1954)
- [Fix] Settings: avoid checking invalid IP (#1981)
- [Fix] Try harder in passing IPv6 addresses
- [Fix] WebUI: use relative path for savemap (#1943)
- [WebUI] Fix message count in throughput summary (#1724)
- [WebUI] Fix NaNs display on Throughput graph
- [WebUI] Restore passwordless login support (#2003)
1.6.5: 22 Oct 2017
- [CritFix] Another portion of tokenization fixes
- [CritFix] Fix memory leak in spf caching logic
- [CritFix] Fix milter commands pipelining
- [CritFix] Fix newlines detection
- [Feature] Filter nan and inf when adding scores
- [Feature] Implement headers flags in mime parser
- [Feature] Support Expires header when using HTTP maps
- [Fix] Actively load skip hashes map in fuzzy storage
- [Fix] Add workaround for IPv6 in sendmail
- [Fix] Authentication Results: Fix SPF smtp.mail_from
- [Fix] Check for magic when checking for an archive
- [Fix] Deal with another case when processing exceptions
- [Fix] Deal with URLs with no slashes after protocol
- [Fix] Do not allow garbadge when checking url domain
- [Fix] Do not ignore short words
- [Fix] Do not strip last character in the last word
- [Fix] Do not treat script content as text
- [Fix] Erase unknown HTML entities
- [Fix] Fix another tokenization issue
- [Fix] Fix DKIM forgeries via multiple headers
- [Fix] Fix emails detection
- [Fix] Fix empty threshold check in greylisting module
- [Fix] Fix enormous scores for R_WHITE_ON_WHITE
- [Fix] Fix loading of per-user redis backend for statistics
- [Fix] Fix multiple headers in DKIM headers list
- [Fix] Fix obscured url in format user@@example.com
- [Fix] Further tokenization fixes
- [Fix] Load skip map from all processes as shared cache is
unavailable
- [Fix] Lowercase words
- [Fix] Milter headers: skip_local / skip_authenticated settings
- [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first
- [Fix] Ratelimit: fix whitelisted_rcpts matching
- [Fix] Some more fixes towards emails detection
- [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't
match
- [Fix] Use greylisting threshold in greylisting module
1.6.4: 10 Sep 2017
- [Feature] Add method to get all content-type attributes in Lua
- [Feature] Add some sanity checks for actions and controller
- [Feature] Allow randomly select User-Agent from a list
- [Feature] Deal with obscured URLs with @ symbols
- [Feature] Milter headers: support adding/removing arbitrary headers
from config
- [Fix] Add another workaround to display history properly
- [Fix] Add missing rspamadm control options to help
- [Fix] Auth-Results: Multiple DKIM signatures
- [Fix] Crash in URL processing
- [Fix] Default monitoring domain for surbl plugin
- [Fix] Detection of maillist optimized and fixed
- [Fix] Do not cache SPF records with PTR elements
- [Fix] Fix blacklists and DMARC in whitelist
- [Fix] Fix exceptions list in surbl
- [Fix] Fix processing of closed tags
- [Fix] Fix PTR processing in SPF
- [Fix] Lowercase HTTP headers to make them searchable from Lua
- [Fix] options.local_networks setting
- [Fix] Ratelimit: lowercase email addresses
- [Fix] Rebalance and slightly rework MX check plugin
- [Fix] Redis script loading in DMARC; URL tags; URL reputation
- [Fix] Reject invalid bh for DKIM signatures earlier
- [Fix] Remove incorrect method `task:set_metric_subject`
- [Fix] Rewriting subjects via force actions module
- [Fix] RPM postinstall
- [Fix] Treat 'rewrite subject' as spam action
- [Fix] Try harder to find urls
- [Fix] Use full URL when making an HTTP request
- [Fix] Use raw urls when sending requests to redirector
- [Fix] Use weight from map for fuzzy scoring
- [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages
1.6.3: 26 Jul 2017
- [CritFix] Fix semicolons parsing in the content type
- [Feature] Add EBL to the default config
- [Feature] Allow to configure monitored
- [Feature] Allow to skip specific hashes in fuzzy storage
- [Feature] Multimap: checking of symbol options
- [Feature] Redis settings: support checking multiple keys
- [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer
- [Fix] Avoid changing content-transfer-encoding header's value
- [Fix] Don't use whitelist/greylist maps as regexp, but as map
- [Fix] Fix get_content method
- [Fix] Header checks: Fix get_raw_header method
- [Fix] Header checks: REPLYTO_UNPARSEABLE rule
- [Fix] Lua_http: freeing
- [Fix] Milter headers: custom headers: removing headers
- [Fix] Parse HREF urls without explicit prefix
- [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the
rule
- [WebUI] Escape strings inside HTML in history
1.6.2: 08 Jul 2017
- [Conf] Remove Rambler email bl for now
- [Conf] Switch RAMBLER_URIBL to a locally managed source
- [CritFix] Switch from ragel to C for Content-Type parsing
- [Feature] Add `-e` option for lua_repl
- [Feature] Add per-domain emails normalisation rules
- [Feature] Add sessions cache to debug dangling sessions
- [Feature] Add short_text_direct_hash for fuzzy check module
- [Feature] Add text_part:get_stats function
- [Feature] Allow to add custom processing script for surbl
- [Feature] Allow to check reply-to email
- [Feature] Allow to customize spam header, remove existing spam
headers
- [Feature] Allow to disable specific workers in the config
- [Feature] Allow to discard messages instead of rejection
- [Feature] Allow to specify custom delimiter in emails plugin
- [Feature] Allow to specify custom User-Agent for rspamc
- [Feature] Allow to store symbols data in Clickhouse
- [Feature] Allow to use HTTPS when connecting to Clickhouse
- [Feature] Enable sessions cache tracking for milter connections
- [Feature] Implement per-line mode in lua_repl (like `perl -p`)
- [Feature] Implement rdns-curve plugin based on rspamd cryptobox
- [Feature] Improve maps cached data lifetime
- [Feature] Improve maps checking frequency
- [Feature] Improve monitored timeouts logic
- [Feature] milter_headers: add `extended_headers_rcpt` option
- [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility
headers
- [Feature] Milter headers: remove-header routine
- [Feature] Multimap: received filters for extracting TLDs from
hostnames
- [Feature] Normalize email aliases in emails module
- [Feature] Re-add rambler email bl (as hashed list)
- [Feature] Reload file maps more frequently
- [Feature] Rework newlines strip parser one more time
- [Feature] Skip updates for messages scanned via controller
- [Feature] Split long DKIM public keys
- [Feature] Store more data when stripping newlines
- [Feature] Support SPF macros transformations
- [Feature] Support suppressing DMARC reports for some domains
- [Fix] Add missing `break` statement
- [Fix] Allow modifiers in SPF macros
- [Fix] DKIM sign tools: edge-cases around use_esld
- [Fix] Do not cache SPF records with macros
- [Fix] Do not overwrite score when setting pre-action
- [Fix] Fix comparison logic
- [Fix] Fix DKIM base64 folding for milter flagged messages
- [Fix] Fix emails module configuration
- [Fix] Fix folding for arc headers when milter interface is used
- [Fix] Fix gmail dots removal
- [Fix] Fix rspamc detection in greylist module
- [Fix] Fix some more issues with HTTP maps
- [Fix] Milter sessions can live forever
- [Fix] Normalize fuzzy probability better
- [Fix] Plug memory leak
- [Fix] RBL: Fixed hashed email address lookups
- [Fix] Try to deal with brain-damaged milter behaviour
- [Fix] Use `\n` to fold headers for milter
- [Rework] Allow to use custom callback for monitored checks
- [Rework] Further steps towards one process monitoring
- [Rework] Send health checks from a single worker
- [WebUI] Round-up throughput summary values
wiz
adam
triaxx
ryoon
schmonz
leot
nia
jperkin
rillig
roy
fhajny
maya