fetchmail-6.3.17 (released 2010-05-06, 25767 LoC):
# SECURITY FIX
* CVE-2010-1167: Fetchmail before release 6.3.17 did not properly sanitize
external input (mail headers and UID). When a multi-character locale (such as
# FEATURES
* Fetchmail now supports a --sslcertfile <file> option to specify a "CA bundle"
file (a file that contains trusted CA certificates). Since these bundled CA
files do not require c_rehash to be run, they are easier to use and immune to
OpenSSL library updates that affect the hash function.
* Fetchmail now supports a FETCHMAIL_INCLUDE_DEFAULT_X509_CA_CERTS
environment variable to force loading the default SSL CA certificate
locations even if --sslcertfile or --sslcertpath is used.
If neither option is in effect, fetchmail loads the default locations.
# REGRESSION FIX
* Fix string handling in rcfile scanner, which caused fetchmail to misparse a
run control file in certain circumstances. Fixes BerliOS bug #14257.
Patch by Michael Banack. This fixes a regression introduced before 6.3.0.
# BUG FIXES
* Plug memory leak when using a "defaults" entry in the run control file.
* Do not print SSL certificate mismatches unless verbose or --sslcertck is
enabled.
* Do not lose "set invisible" in fetchmailconf. (Michael Barnack)
# CHANGES
* Usability: SSL certificate chains are fully printed in -v -v mode, and there
are now helpful pointers to --sslcertpath and c_rehash for "unable to get
local issuer certificate" and self-signed certificates -- these usually hint
to missing root signing CAs in the certs directory.
* Several fixes for compiler (GCC, Intel C++, CLang) and autotools warnings
* Memory allocation failures will now cause abnormal program abort (SIGABRT),
no longer an exit with unspecified code.
# DOCUMENTATION
* Fix table of global option to read "set softbounce" where there used to be a
2nd copy of "set spambounce". Patch by Michael Banack, BerliOS Bug #17067.
* In the --sslcertpath description, mention that OpenSSL upgrade (and a 0.9.X
to 1.0.0 upgrade in particular) may require running c_rehash.
# TRANSLATION UPDATES
[zh_CN] Chinese/simplified (Ji Zheng-Yu)
[cs] Czech (Petr Pisar)
[nl] Dutch (Erwin Poeze)
[fr] French (Fr\xc3<A9>d\xc3<A9>ric Marchal)
[de] German
[id] Indonesian (Andhika Padmawan)
[it] Italian (Vincenzo Campanella)
[ja] Japanese (Takeshi Hamasaki)
[pl] Polish (Jakub Bogusz)
[sk] Slovak (Marcel Telka)
[vi] Vietnamese (Clytie Siddall)
# KNOWN BUGS AND WORKAROUNDS:
(this section floats upwards through the NEWS file so it stays with the
current release information - however, it was stuck with 6.3.8 for a while)
* fetchmail does not handle messages without Message-ID header well
(See sourceforge.net bug #780933)
* BSMTP is mostly untested and errors can cause corrupt output.
* Sun Workshop 6 (SPARC) is known to miscompile the configuration file lexer in
64-bit mode. Either compile 32-bit code or use GCC to compile 64-bit
fetchmail. Note that fetchmail doesn't take advantage of 64-bit code,
so compiling 32-bit SPARC code should not cause any difficulties.
* fetchmail does not track pending deletes over crashes
* the command line interface is sometimes a bit stubborn, for instance,
fetchmail -s doesn't work with a daemon running
fetchmail-6.3.16 (released 2010-04-06, 25574 LoC):
# BUG FIX
* Fix --interface option, broken in 6.3.15. Reported by Vladmimir Stavrinov.
Fixes Debian Bug #576717.
# CHANGE
* Call OpenSSL_add_all_algorithms(). This is needed to support non-mandatory
and non-standard algorithms in certificates.
Sjoerd Simons, to fix Debian Bug #576430.
OpenSSL 0.9.8* does not load - for instance - the SHA256 digest by default.
Reported as OpenSSL RT#2224.
fetchmail-6.3.15 (released 2010-03-28, 25572 LoC):
# FEATURE
* Fetchmail now supports a bad-header command line or rcfile option that takes
exactly one argument, accept or reject (default). This specifies how messages
with bad headers retrieved from the current server are to be treated.
# BUG FIXES
* In the rcfile, recognize "local" as abbreviation for "localdomains", as
documented. The short form has not ever worked since this feature was added in
January 1997. Reported by Fr\xc3<A9>d\xc3<A9>ric Marchal.
* Do not close stdout when using mda and "bsmtp -" at the same time.
* Log operating system errors when BSMTP writes fail.
* Fix verbose mode progress formatting regression from 6.3.10; SMTP trace lines
were no longer on a line of their own. Reported by Melchior Franz.
* Check seteuid() return value and abort running MDA if switch fails.
* Set global flags in a consistent manner. Make --nosoftbounce and
--nobounce work from command line (these used to work in rcfiles).
Reported and fix confirmed working by N.J. Mann. (Sunil Shetye)
* Properly import h_errno declarations, even on systems where h_errno isn't a
macro. (Adds ./configure check, fixes Cygwin dllimport warnings.)
# CHANGES
* The repository has been converted and moved from the Subversion (SVN) format
kindly hosted by Graham Wilson over the past years to Git format hosted on
Gitorious.org. My deepest thanks to Graham Wilson for this service that
kept us going when BerliOS's Subversion service was faulty in its early days.
* This opportunity was used to convert BRANCH_6-2 and BRANCH_1-9-9 to
GnuPG-signed tags, as a sign that these are now closed.
* The outdated SVN trunk is now called "oldtrunk" in Git just to save the work
for future reference. All development in the past few years was on BRANCH_6-3.
* master was branched from BRANCH_6-3. BRANCH_6-3 is now obsolete (and in fact
was also converted to a tag to record where the conversion from SVN to Git
took place).
* "make check" now skips HTML validation if xmllint or XHTML DTD are missing.
# DOCUMENTATION
* Web site and documentation were adjusted to reflect the SVN->Git move.
* The fetchmail manual page is now much clearer on the user id switching
(seteuid) when using --mda while running as the super user.
# TRANSLATION UPDATES, by language name
* [zh_CN] Chinese (Simplified), by Ji Zheng-Yu
* [cs] Czech, by Petr Pisar
* [nl] Dutch, by Erwin Poeze
* [fr] French, by Fr\xc3<A9>d\xc3<A9>ric Marchal
* [de] German
* [id] Indonesian, by Andhika Padmawan
* [it] Italian, by Vincenzo Campanella
* [ja] Japanese, by Takeshi Hamasaki
* [pl] Polish, by Jakub Bogusz
* [vi] Vietnamese, by Clytie Siddall
- change to how getmail counts messages in an IMAP mailbox; prevents problems
where getmail would only see the first 500 messages in a mailbox with some
IMAP servers that return oddball responses to SELECT. Thanks: David Damerell.
The main thing that this new release adds is full support for the spamtest,
spamtestplus and virustest extensions. These extensions implement a uniform
means of testing the content spam/virus status headers in mail messages:
http://ietfreport.isoc.org/idref/rfc5235/
Not many people have tested these new features yet and therefore these are
currently considered experimental. Test these thoroughly before you start using
them! Documentation is available in share/doc/dovecot/spamtest-virustest.txt
Changelog Sieve v0.1.16:
* Finished implementation of spamtest, spamtestplus and virustest
extensions. These are not enabled by default and need to be
activated with the sieve_extensions setting. Documentation available
in doc/spamtest-virustest.txt
+ Vacation extension: the from address of the generated reply is now
by default equal to whatever known recipient alias matched the
headers of the message. If it is one of the aliases specified with
:addresses, it is used in stead of the envelope recipient address
that was used before.
+ Restructured and optimized the lexical scanner.
+ Added --with-docs configure option to allow disabling installation
of documentation.
- Accidentally omitted 'extern' in two declarations of global
variables in header files, causing compile failures on certain
systems.
- Deprecated imapflags extension: fixed implicit assignment of flags.
Turns out this never really worked, but the effect of this bug was
obscured by the removeflag bug fixed in the previous release.
- Fixed various memset argument mixups in enotify extension. This
caused warnings on certain systems, but luckily no adverse effects
at runtime.
Changelog:
Implemented Features:
* Added Mail_mime::setContentType() function with possibility to set various
types in Content-Type header (also fixes problem with boundary parameter
when Content-Type header was specified by user) [alec]
1.) Always install the binary. The permissions of "/var/mail" on
the build system are just no good indicator to determine whether
this program is required or not.
2.) Don't try to set the permissions of the binary in the makefile
and add it to "SPECIAL_PERMS" instead. Mark the package as capable
of user destinaton directory installation again.
Bump package revision because the binary package has changed at least
under NetBSD with default permissions for "/var/mail".
1.3.1 09-February-2010
---------------------------------------------
- Fix D-Bus connection leak in Nepomuk search backend.
- Disable slow query logging by default for internal MySQL.
1.3.0 20-January-2010
---------------------------------------------
- Work around D-Bus bug that could cause SEARCH to hang.
1.2.90 06-January-2010
---------------------------------------------
- Fix change notifications for search results.
- Fix database creation with PostgreSQL.
- Fix copying of item flags.
- Fix internal MySQL shutdown.
- Support PostgreSQL in internal mode.
- Fix table name case mismatch.
1.2.80 01-December-2009
---------------------------------------------
- Support for collection content type filtering as part of LIST.
- Adapt to Nepomuk query service changes.
- Experimental support for PostgreSQL.
- Support for preprocessor agents.
- Support for distributed searching.
- Support for agents creating virtual collections.
- Protocol parser fixes for non-Linux/non-KDE clients.
- Support for single-shot searches using the Nepomuk query service.
- Support HRID-based LIST operations.
- Support RID-based MOVE, COLMOVE, LINK and UNLINK opertions.
- Respect cache-only retrieval also regarding on-demand syncing.
- Add configuration accepted/rejected signals to the agent interface.
- Fix change notification compression when using modified parts sets.
- Use one retrieval pipeline per resource.
- Reduce unecessary change notification on flag changes.
- Fix RID quoting.
- Fix resource creating race for autostarted agents.
- Create new database also when using external db servers.
- Return the created result collection when creating a persistent search.
milter manager is a milter to use multiple milters effectively.
If milter manager is introduced, milter manager administrates milters instead of
MTA. The was has some advantages:
* reduce milter administration cost
* combine milters flexibly
* The random scrolling problem on selecting folders with GTK+ 2.12 or later
was fixed.
* The problem that config files are not updated after creating new accounts
was fixed.
* The problem that folder config file was not updated after changing IMAP
or News account name was fixed.
* Several typos in some dialogs were fixed.
* The Japanese manual was updated.
This release of ClamAV introduces new malware detection mechanisms and other
significant improvements to the scan engine. The key features include:
- The Bytecode Interpreter: the interpreter built into LibClamAV allows
the signature writers to create and distribute very complex detection
routines and remotely enhance the scanner's functionality
- Heuristic improvements: improve the PE heuristics detection engine by
adding support of bogus icons and fake PE header information. In a
nutshell, ClamAV can now detect malware that tries to disguise itself
as a harmless application by using the most common Windows program icons.
- Signature Improvements: logical signature improvements to allow more
detailed matching and referencing groups of signatures. Additionally,
improvements to wildcard matching on word boundaries and newlines.
- Support for new archives: 7zip, InstallShield and CPIO. LibClamAV
can now transparently unpack and inspect their contents.
- Support for new executable file formats: 64-bit ELF files and OS X
Universal Binaries with Mach-O files. Additionally, the PE module
can now decompress and inspect executables packed with UPX 3.0.
- Support for DazukoFS in clamd
- Performance improvements: overall performance improvements and memory
optimizations for a better overall resource utilization experience.
- Native Windows Support: ClamAV will now build natively under Visual
Studio. This will allow 3rd Party application developers on Windows
to easily integrate LibClamAV into their applications.
The complete list of changes is available in the ChangeLog file. For upgrade
notes and tips please see: https://wiki.clamav.net/Main/UpgradeNotes096
Pkgsrc changes:
- Removed most of the package options; using the options framework for
those choices was not quite correct in the first place. Some have now
fixed values (Perl warnings and taint checks always enabled), some
got converted to variables settable from the make command line, see
options.mk.
- pkglint complained about the variable PLIST_ADD so I renamed it to
DYNAMIC_PLIST.
- SpamAssassin does not come with rules anymore. As a starting point
the official archive of rules at the time of the SA release is
included. At installation time this set of rules gets installed
through the pkgsrc INSTALL file.
- Removed patch-ba and patch-bc, both were integrated upstream.
- Removed patch-be. The quick fix for the bad rule was replaced by
a permanent solution.
Summary of major changes since 3.3.0
====================================
bug 6335: add Spamhaus DBL as URIBL_DBL_SPAM rule
Bug 6370: update ImageInfo plugin to latest release
bug 6215, bug 6294: RCVD_IN_CSS rule was broken. the check_rbl_sub() syntax
was incorrect, resulting in missing hits
bug 6361: list 2tld and 3tld sub-domain hosters for URIBL/SURBL/DBL queries;
NOTE for SARE users: This file replaces the SARE file
http://www.rulesemporium.com/rules/90_2tld.cf, which will be deprecated as from
2010-05-01.
Bug 6369, 6356, 6373: WIN32 support for spamd improved
Bug 6267: Solaris 10 requires --syslog-socket=native
bug 6304 spamd is spawning and killing processes too often - Added spamd
adjustments to info level and more information for administrators + small fix
to Makefile.PL
Bug 6310: sa-learn --import gives Insecure dependency in open
Bug 6313: -Q or -q AND -x should not result in creation of a ~/.spamassassin
dir; plus: taint issues fixed
Bug 6342: make test failure on if_can under perl 5.6
Bug 6340: Impossible to find user home directory of VPOPMAIL alias
Bug 6072, 6343: POD warnings, documentation fixes
Bug 6304 (trivial), reduce sysadmin's stress level by lowercasing
the 'INTERRUPTED' in a logged message:
spamd: handled cleanup of child pid [...] due to SIGCHLD: INTERRUPTED
Bug 6329: POSIX::strftime in call under Win32 ActivePerl causes Perl to hang up;
formatting option %e is not in a POSIX standard, use %d instead and edit
Bug 6322: In DKIM ADSP eval test check_dkim_adsp() the '*' is handled incorrectly
Bug 6327: Fix calling argument in utility used to determine DCC's homedir
Bug 6316: DCC.pm, wrong options for dcc_proc, (plus: avoid a warning on undef
in logger when dccifd socket is not provided)
Bug 6287: improved DKIM plugin debugging
Bug 6321 - _TOKENSUMMARY_ not working in 3.3.0 (Plugin/Bayes.pm looks-up a tag
from wrong location)
Bug 6312 - uninitialized value $start_time in spamd
bug 5761: trivial doc fix: document SPAMD_LOCALHOST test-control env variable
Summary of major changes since 3.2.5
====================================
COMPATIBILITY WITH 3.2.5
- rules are no longer distributed with the package, but installed by
sa-update - either automatically fetched from the network (preferably)
or from a tar archive, which is available for downloading separately
(see below, section INSTALLING RULES);
- CPAN module requirements:
- minimum required version of ExtUtils::MakeMaker is 6.17;
- modules now required: Time::HiRes, NetAddr::IP (4.000 or later),
Archive::Tar (1.23 or later), IO::Zlib;
- minimal version of Mail::DKIM is 0.31 (preferred: 0.37 or later);
expect some tests in t/dkim2.t to fail with versions older than 0.36_5;
- no longer used: Mail::DomainKeys, Mail::SPF::Query;
- either Digest::SHA or the older Digest::SHA1 is required, though
note that the DKIM plugin requires Digest::SHA for sha256 hashes
and Razor agents still need Digest::SHA1;
- some IPv6 functionality requires IO::Socket::INET6;
- if keeping the AWL database in SQL, the field awl.ip must be extended to
40 characters. The change is necessary to allow AWL to keep track of IPv6
addresses which may appear in a mail header even on non-IPv6 -enabled host.
While at it, consider also adding a field 'signedby' to the SQL table 'awl'
(and adding 'auto_whitelist_distinguish_signed 1' to local.cf);
see sql/README.awl for details. The change need not be undone even if
downgrading back to 3.2.* for some reason;
- fixing a protocol implementation error regarding a PING command required
bumping up the SPAMC protocol version to 1.5. Spamd retains compatibility
with older spamc clients. Combining new spamc clients with pre-3.3 versions
of a spamd daemon is not supported (but happens to work, except for the
PING and SKIP commands);
- if using one of the plugins (FreeMail, PhishTag, Reuse) which were
previously not part of the official package, please retire your local copy
to avoid it conflicting with a new native plugin;
- as the plugin AWL is no longer loaded by default, to continue using it
the following line is needed in one of the .pre files (e.g. local.pre):
loadplugin Mail::SpamAssassin::Plugin::AWL
- it may be worth mentioning that a rule DKIM_VERIFIED has been renamed
to DKIM_VALID to match its semantics;
- the DKIM plugin is now enabled by default for new installs, if the perl
module Mail::DKIM is installed. However, installation of SpamAssassin
will not overwrite existing .pre configuration files, so to use DKIM when
upgrading from a previous release that did not use DKIM, a directive:
loadplugin Mail::SpamAssassin::Plugin::DKIM
will need to be uncommented in file "v312.pre", or added to some
other .pre file, such as local.pre;
- due to changes in some internal data structures (like Bug 6185, 6254),
some third-party plugins may need to be updated. One such example is
the ClamAVPlugin plugin - please find a fresh version, which can be used
with both SpamAssassin versions 3.2.5 and 3.3.0, on its wiki page at
http://wiki.apache.org/spamassassin/ClamAVPlugin
- versions of amavisd-new between 2.5.2 and 2.6.1 (inclusive) are incompatible
with SpamAssassin 3.3; please upgrade amavisd to 2.6.2 or later, or apply
a workaround https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6257
- support for versions of perl 5.6.* is being gradually revoked
(may still work, but no promises and no support);
- preferred versions of perl are 5.8.8, 5.8.9, and 5.10.1 or later;
- on FreeBSD, please avoid using multithreaded versions of perl older
than 5.10.0 due to small default main thread's stack size, which may
not suffice for some regular expression evaluations;
INSTALLING RULES
Rules are normally installed by running a sa-update command.
The version of sa-update program should match the version of SpamAssassin
modules, so invoking sa-update should be performed only after installing
or upgrading SpamAssassin code, not before.
Installing rules from network is done with a single command,
normally run as root:
sa-update
Installing rules from files:
obtain all the following files:
Mail-SpamAssassin-rules-xxx.tgz
Mail-SpamAssassin-rules-xxx.tgz.asc
Mail-SpamAssassin-rules-xxx.tgz.md5
Mail-SpamAssassin-rules-xxx.tgz.sha1
(where xxx may look something like '3.3.0.r893295')
install rules from a compressed tar archive:
sa-update --install Mail-SpamAssassin-rules-xxx.tgz
(sa-update will need corresponding .asc and .sha1 files with the
same base name in the same directory as the .tgz file)
MAIN NEW FEATURES
- IPv6 support was substantially improved (see below);
- many improvements to the DKIM plugin (understands author domain signatures,
supports multiple signatures, ADSP support with overrides) - (see below);
- added 'if can(Class::method)' conditional statement, allowing configuration
settings to be conditional on plugin capabilities without requiring
new version releases to do so;
- added a --verbose option to the sa-update utility to show updated channels;
- added a configuration option 'time_limit', defaulting to 300 seconds
or whatever the caller (like spamd) provides; attempting to gracefully
terminate the checking when a time limit is reached, reporting the score
and test hits that were collected so far, along with an added hit on
a rule TIME_LIMIT_EXCEEDED;
- more expensive code sections are now instrumented with timing measurements;
timing report is logged as a debug message by the end of processing,
and made available to a caller and to 'add_header' directives through
a TIMING tag;
- added a configuration option skip_uribl_checks to the URIDNSBL plugin,
cross-documented it with skip_rbl_checks;
- preserve order of declared 'add_header' header fields;
- configurable network mask length for the AWL plugin (see below);
- added support for DCC reputations (see below);
- improved error handling and robustness (see below);
- added timestamps when logging on stderr;
- allowed debug areas to be excluded from debugging,
e.g.: -D all,norules,noconfig,nodcc
BUILDING AND PACKAGING
- rules are no longer distributed with the package, but installed by
sa-update
- Makefile.PL has been simplified and a bug fixed in a DESTDIR support
by increasing the minimum required version of ExtUtils::MakeMaker to 6.17
- tools check_whitelist and check_spamd are now included in the distribution,
now called 'sa-awl' and 'sa-check_spamd'
WORKAROUNDS TO PERL BUGS AND LIMITATIONS
- modified the Check.pm plugin to produce smaller chunks of source code
from rules (60 kB) to avoid Perl compiler crashing on exceeding stack size;
- localized global variables $1, $2, etc at several places, avoiding taint
issue from propagating;
- avoided Perl I/O bug by replacing line-by-line reading with read() where
suitable, or played down the EBADF status in other places and only report
it as a dbg instead of a die - while also providing a little speedup
(10 .. 25 %) on reading a message;
- provided a new sub Message::split_into_array_of_short_lines to split
a text into array of paragraph chunks of sizes between 1 kB and 2 kB,
giving less opportunity to runaway regular expressions in rules;
fixes bugs: 5717, 5644, 5795, 5486, 5801, 5041;
MEMORY FOOTPRINT
- as a side-effect of compiling rules in smaller chunks (to avoid compiler
crashes), virtual memory footprint of SpamAssassin is reduced;
- saved some memory by not importing the Pod::Usage unless it is needed;
- saved 350k+ of memory in sa-compile by replacing DynaLoader with XSLoader;
- removed unneeded index from MySQL bayes_token table;
IPv6 SUPPORT
- added IPv6 support for trusted_networks, internal_networks, msa_networks,
whitelist_from_rcvd, and other stuff that uses NetSet and the Received
header field parser, using NetAddr::IP;
- allowed usage of a remote dccifd host through an INET or INET6 socket;
- added IPv6 support to AWL plugin and its utility modules; a network
mask length is now configurable and defaults to /48, which controls
what data is stored in an AWL database;
- sql/README.awl and sql/awl_*.sql: increased suggested awl.ip field width
to 40 characters to be able to hold IPv6 addresses;
- IP_PRIVATE now includes ipv6 variants of private address space,
as well as the ipv6-mapped ipv4 addresses.
- NetSet now understands that ::ffff:192.168.1.2 and 192.168.1.2 are
the same address;
- IPv6 addresses are now properly read from Received header fields;
- when reading Received header fields, the "IPv6:" prefix is stripped from
IPv6 addresses, and "::ffff:" is removed from IPv6-mapped IPv4 addresses
(so strings can match them as simply IPv4 addresses);
- ::1/128 is always included in the trusted_networks/internal_networks set
similar to 127.0.0.0/8;
- some of the IPv6 functionality in SpamAssassin requires that a perl module
IO::Socket::INET6 is available (like accessing a DNS resolver over inet6,
talking to a dccifd host over inet6 socket, SPAMC protocol);
SPAMC
- Mail::SpamAssasin::Client ping may erroneously result in broken pipe;
bump spamc protocol version to 1.5, updated spamd, spamc and Client.pm;
- added -n / --connect-timeout switch to spamc, allowing to separate
a connection timeout from communication timeout;
- added --filter-retries and --filter-retry-sleep;
- increased allowed line length in spamc.conf files to 8 KiB and report
an error when the limit is exceeded;
- fixed issue where spamc would not time out connections to a hung spamd;
- spamc client library leaked the zlib compression buffer if compression
is used;
- spamc long option '--dest' was broken;
SPAMD
- when spamd is started with the daemonize option do not exit the parent
until a child signals that it has logged the pid, to allow a wrapper
script to simply continue immediately after starting spamd;
- additional tempfile cleanup in kill_handler;
- added SPAMD_LOCALHOST option to "make test" to allow specifying
non-127.0.0.1 IP address for use in FreeBSD jail;
API
- adding one optional argument to Mail::SpamAssassin::parse allows caller
to pass additional out-of-band information to SpamAssassin (such as a
deadline time, DKIM verification results, information about a SMTP session,
or dynamic rule hits); this information is made available to plugins and
the rest of the code through a 'suppl_attrib' hash;
- added option 'master_deadline' to the suppl_attrib argument of a
Mail::SpamAssassin::parse method, allowing the caller to override a
time_limit configuration setting;
- Plugin::Check - pick up 'rule_hits' from caller via the new mechanism
and call got_hit() on them;
- simplified adding dynamic score hits and dynamic rules by plugins
(such as AWL, CRM114, FuzzyOcr, Check) by letting got_hit() accept
options tflags and description, and letting it store a supplied
dynamic score for proper reporting;
- let the timing breakdown information be accessible to a caller through
the existing get_tag mechanism (tag TIMING);
- let the generated header fields ('add_header' configuration options)
be accessible to a caller through the existing get_tag mechanism
(tags ADDEDHEADER, ADDEDHEADERHAM, ADDEDHEADERSPAM);
RULES
- rules are no longer distributed with the package;
- new scores were generated by a genetic algorithm (GA) and then manually
tweaked based on cleaned datasets supplied by a dozen volunteers;
- dropped redundant rules or rules causing too many false positives;
- added or updated many rules; incomplete list in no particular order:
vbounce, lotsa_money, muchmoney, image spam, fill_this_form, FreeMail,
European Parliament, HTML attachments, uri_obfu*, urinsrhsbl, urinsrhssub,
urifullnsrhsbl, URI_OBFU_X9_WS, rDNS=localhost, INVALID_DATE_TZ_ABSURD,
RCVD_IN_PSBL, FRT_VALIUM*, BOUNCE_MESSAGE, VBOUNCE_MESSAGE,
__BOUNCE_UNDELIVERABLE, HELO_STATIC_HOST, FILL_THIS_FORM_FRAUD_PHISH,
CHALLENGE_RESPONSE, DKIM_VALID, DKIM_VALID_AU, DKIM_ADSP_*,
NML_ADSP_CUSTOM_{LOW,MED,HIGH}, __VIA_ML, MIME_BASE64_TEXT, LOTTO_URI,
FORGED_MUA_THEBAT_BOUN, FORGED_MUA_THEBAT_CS, UNRESOLVED_TEMPLATE,
__THEBAT_MUA, __ANY_OUTLOOK_MUA, RP_MATCHES_RCVD, one-word X-Mailer,
SPAN rules, skype and misquoted-HTML rules, HTML obfuscation and
Google feedproxy URI rules, advance_fee updates including further
evolved advance fee second-order metarules, test rule for
postmaster+abuse missing, FROM_MISSPACED, fixed FROM_CONTAINS_TAB, a
Facebook redirector pattern, fixed FPs with TVD_SPACE_RATIO regarding
one-word emails and ISO-2022-JP, added exclusion for __ISO_2022_JP_DELIM
to OBFUSCATING_COMMENT, GAPPY_SUBJECT, PLING_QUERY and FM_FRM_RN_L_BRACK
rules, RATWARE_BOUNDARY plus variant, superseded all previous
RATWARE_OUTLOOK stuff, resolved FP in obfuscated URI rule, fixed breakage
in tbird image rule, fixed SUBJECT_FUZZY_MEDS FP on unobfuscated "meds",
added misspaced From header field rule, numeric+cctld URI rule,
updated FH_DATE_PAST_20XX, ...
- added PSBL blacklist - http://psbl.surriel.com/
- added support for http://www.spamhaus.org/css/
- replaces HABEAS, BSP and SSC with RP CERTIFIED;
- use ReturnPath's RNBL, replacing SSBL;
- added rule for plain text attachments with octet-stream MIME type;
- avoided false positives on ISO-2022-JP messages in several rules;
- removed massmailers from uridnsbl_skip_domain in 25_uribl.cf;
- updated various default whitelists, uridnsbl_skip_domain, adsp_override, ...
PLUGINS
- new plugins: FreeMail, PhishTag, Reuse;
- now enabled by default: DKIM;
- now disabled by default: AWL;
- retired plugin: DomainKeys;
AWL PLUGIN
- plugin AWL is now disabled by default;
- added new configuration options auto_whitelist_ipv4_mask_len and
auto_whitelist_ipv6_mask_len to allow more control on what part of
an IP address is stored into an AWL database;
- README.awl: increased a suggested awl.ip field width to 40 characters
to support IPv6 addresses;
- AutoWhitelist.pm: allowed storing a canonicalized IPv6 address, cropped
to a configurable network mask (previously causing SQL server errors:
'value too long');
- let AWL with SQL keep separate records for DKIM-signed and unsigned mail
(when auto_whitelist_distinguish_signed configuration option is true,
and a field awl.signedby exists);
- avoided a race condition in SQLBasedAddrList.pm when multiple processes
try to insert-or-update an awl SQL record: trying INSERT first, and if
that fails go for UPDATE;
- gracefully handle NaN from corrupted database or a broken emulator or
virtualizer;
DCC PLUGIN
- added support for DCC reputations, added setting dcc_rep_percent,
new test check_dcc_reputation_range(), new tag DCCREP
(DCC servers supply reputation data only to licensed clients);
- allowed usage of a remote dccifd host through an INET or INET6 socket;
DKIM PLUGIN
- the DKIM plugin is now enabled by default for new installs if the perl
module Mail::DKIM is installed. However, installing SpamAssassin will
not overwrite existing .pre configuration files, so to use DKIM when
upgrading from a previous release that did not use DKIM, the directive:
loadplugin Mail::SpamAssassin::Plugin::DKIM
will need to be uncommented in file "v312.pre", or added to some
other .pre file, such as local.pre;
- absolute minimal version of Mail::DKIM is 0.31;
support for ADSP requires Mail::DKIM 0.34;
a DNS test (and rule) for NXDOMAIN is operational since Mail::DKIM 0.36_5,
so effectively the recommended version is Mail::DKIM 0.37 or later;
- a perl module Digest::SHA is required if the DKIM plugin is enabled.
If a perl module Digest::SHA is available, the module Digest::SHA1
becomes optional as far as SpamAssassin is concerned, but is still
needed by Razor agents;
- added support for multiple signatures (useful for whitelisting);
- plugin now distinguishes author domain signatures from third party
signatures (useful for whitelisting);
- provides a tag DKIMIDENTITY (in addition to DKIMDOMAIN);
- DKIM now supports Author Domain Signing Practices - ADSP (RFC 5617);
- use the Mail::DKIM::AuthorDomainPolicy instead of Mail::DKIM::DkimPolicy,
when available (since Mail::DKIM 0.34);
- implements an 'adsp_override' configuration directive and adds
an eval:check_dkim_adsp check, which is used by new DKIM_ADSP_* rules;
- rules contain an initial set of 'adsp_override' directives, listing
some of the more popular target domains for phishing (applicable only to
domains which sign all their direct mail with a DKIM or DK signature);
- this plugin can now re-use Mail::DKIM verification results if made
available by a caller, which saves resources and makes it possible
for SpamAssassin to work on a truncated large mail without breaking
DKIM signatures;
- check_dkim_signed and check_dkim_adsp eval rules can now take an optional
list of domain names, which limits their action to listed domains only.
It facilitates building DKIM-based rules for specific domains, without
having to resort to meta rules;
- draft-ietf-dkim-ssp-10/RFC-5617 made Author Domain Signature based on 'd':
updated ADSP code accordingly; changed whitelisting code to be based on
SDID ('d') instead of AUID ('i');
- Plugin/DKIM.pm: terminology changes in comments and logging according
to RFC 5617 and draft-ietf-dkim-rfc4871-errata-07;
BUG FIXES
- fixed Rule2XSBody segfaults;
- no longer treat user data as perl booleans (a string "0" is a false);
- avoid data from the wild be interpreted as perl regular expressions;
- ArchiveIterator: prevent _scan_directory from passing directories
to _scan_file (on NFS it would fail with EISDIR on read(2);
- fixed inserting the SpamAssassin -generated header fields after a
multiline Return-Path header field;
- fixed vpopmail support;
- fixed incorrect mode bits when creating lock files for AWL;
- fixed some cases where :addr headers were parsed incorrectly;
- fixed leakage of 'whitelist_from_rcvd' entries between spamd users;
- fixing run_and_catch, which failed to catch a non-timed run;
- 127/8 isn't an illegal IP;
- reworked the M::S::Timeout module to deal with nested timers as one would
expect: an inner timer shouldn't be able to extend an outer timer's limit;
account for time elapsed in the submitted subroutine when restarting an
outer timer; reset() should have accounted for time already spent;
deal with nested timed runs where alarm(0) does not provide remaining time;
- the 'exists:' evaluator in HEADER rules now works as documented
and tests for existence of a header field, instead of testing for
a header field body being nonempty; internally, the pms->get can
also now distinguish between empty and nonexistent header fields;
- applied fixes to header fields parsing in several places: header field
names are case-insensitive, whitespace is not required after a colon,
obsolete rfc822 syntax allowed whitespace before a colon;
VBounce: match "Received:" only at the beginning of a line;
- fixed bugs 6237 and 6295: 1.0.0.0/8 and 2.0.0.0/8 are now valid allocated
address ranges, fixed a corresponding rule RCVD_ILLEGAL_IP;
- fixed bug 6205 comment 5 in URIDetail.pm;
- 'pyzor_options' in Plugin/Pyzor.pm was not untainted;
- made the URIDetail plugin taint safe;
- fixed parsing of multi-line Received header fields for
BOUNCE_MESSAGE/VBOUNCE_MESSAGE et al;
- Bug 6206, Bug 2536: spamd: untaint directory as obtained from a password
file or from vpopmail utilities, avoid implicit untainting; report error
if user preferences file exists but cannot be accessed;
- avoided using raw data from DNS as a regexp in Plugin/ASN.pm;
- ensured the dbg() and info() calls always return the same value (true)
regardless of log level;
- suppressed logging of $& when its value is not available (i.e. when
no regexp has been evaluated during rule evaluation);
- Exporter never really worked in SA, was not enclosed in BEGIN {};
- masses/runGA and masses/mk-baseline-results: prevent a shell 'source'
command from loading an unrelated file named 'config' which happens to be
in the current PATH - must use a ./ in an arg to a 'source' command;
ERROR HANDLING, ROBUSTNESS
- improved error detection and reporting: test status of all system calls
and I/O operations (or explicitly document where not), and report
unexpected failures;
- eval calls now check for eval result instead of testing the $@, which
is not always reliable;
- localized $@ and $! in DESTROY methods to prevent potential calls to eval
and calls to system routines in code executed from a DESTROY method
from clobbering global variables $@ and $!;
- Util::helper_app_pipe_open_unix: contain a failing exec with an eval
to prevent additional cases of process cloning. The exec could fail
this way when given tainted arguments;
- Util::helper_app_pipe_open_unix: flush stdout and stderr before forking,
otherwise an error reported by exec (such as 'insecure dependency')
was lost in a buffer;
- eval-protected an open($fh,'-|') to capture implied fork failures
due to lack of system resource;
- explicit untainting: combine "use re 'taint'" with untaint_var(),
avoiding implicit perl untainting, along with workarounds to prevent it;
- added 'use strict' where missing;
- avoided a bunch of warnings on "Use of uninitialized value";
- clearly report reasons for helper application process failures;
- t/SATest.pm: provide information about the process failure reason
if a system() call fails; improved its reporting of failures;
- improved error reporting in Plugin/DCC.pm on finding a DCC home directory
to facilitate troubleshooting;
OTHER CHANGES
- pseudoheader "ALL:raw" returns a pristine header section,
and pseudoheader "ALL" returns a cleaned header section
- total rewrite of URI detection in plain text body;
- many updates to the list of top level domains;
- added 'util_rb_3tld', allowing 3-level TLDs to be listed in URIBLs and
allowing new 3TLDs to be added from rule updates;
- avoided trusted_networks bog down due to O(n^2) loop with millions
of entries;
- applied fixes to Plugin/VBounce.pm, updated VBounce ruleset;
- added support for a 'Communigate Pro' Received header field;
- parse Communigate Pro "with HTTPU" auth token;
- let DependencyInfo.pm understand a concept of recommended module version,
besides a required version;
- provided a workaround for Net::DNS::Packet::new inconsistency;
- let SpamAssassin use either Digest::SHA or Digest::SHA1, whichever is
available (the Digest::SHA is now a base module since perl 5.10.0);
- improved parsing of eval-type rules: allow unquoted domain names as
arguments, disallow unmatched quotes;
- provided a new module Mail::SpamAssassin::BayesStore::BDB. It should be
treated as alpha-quality (needs more testing) and is not yet ready for
production use;
- exposed existing function 'received_within_months' as an eval function
in Plugin/HeaderEval.pm;
- moved rc script to /var/lock/subsys/spamd instead of
/var/lock/subsys/spamassassin so 'service spamd status' will work;
- added feature to re-download MIRRRORED.BY files at least once a week, or if
'sa-update --refreshmirrors' switch is used;
- input delimiter $/ can be corrupted by a plugin, localize $/ and $\ before
calling a plugin;
- bumped the retry counter to 180 seconds for starting spamd on slow machines;
- resolved Bug 5325: syslog severity level in spamc/libspamc.c for max
message size (changed LOG_ERR into LOG_NOTICE for the message:
"skipped message, greater than max message size");
- added checker to avoid taint warnings if hostname is returned as '(none)';
- altered sa-update to produce an error message if a channel doesn't exist;
- Bug 6150, Bug 6127, Bug 5981, Bug 5950, Bug 6191: let spamd log/report
a child process exit status or aborting condition in an informative way;
- added checker to detect accidental match-everything regexps in rules;
- updated garescorer for 3.3.0: use more epochs in GA runs for better scores;
clarify some mass-check warning output, ensure rule name always appears at
start of line; if a rule had no default/existing score in 50_scores.cf,
don't tell the GA that 1.0 is an appropriate default value, instead pick
the midway point of its score range. this produces better results;
remove some dead code from masses/score-ranges-from-freqs;
- set garescorer.c to report performance as iterations per second;
- added test to ensure that all config settings are correctly handled when
switching between users; added more config setting type metadata to enable
those tests to work; and fix URIDetail to store config on the {conf} object,
not on the plugin;
- moved 'release tests' to xt/ directory; mirror long-running, net-tests and
stress tests with xt/50_testname.t scripts to enforce their run before a
release;
- made numerous additional and updated self-tests;
- added a Test::Perl::Critic release-test;
- cleaned up some code based on suggestions by perl module Test::Perl::Critic,
among others:
. enable TestingAndDebugging::ProhibitNoStrict test but allow the
use of 'no strict "refs"';
. deal with BuiltinFunctions::RequireGlobFunction;
. deal with ControlStructures::ProhibitMutatingListFunctions
removing this exception from xt/60_perlcritic.t;
. deal with BayesStore/BDB.pm, Variables::ProhibitConditionalDeclarations
. now that the module Time::HiRes is a required module, we can afford
to replace a select() with Time::HiRes::sleep, and remove exception
BuiltinFunctions::ProhibitSleepViaSelect from xt/60_perlcritic.t;
- updated documentation, fixing numerous typos and mistakes in documentation
text and in log messages;
- extensively improved development process:
. automated testing through Hudson, a continuous integration tool;
. improved mass-check system and rules oversight;
