Vendor update message (excerpt):
Many thanks to Julien ÉLIE for preparing this release.
Special notes:
The way checkpoints are handled by innreport for innd and innfeed has totally changed to provide more accurate daily statistics. The first Usenet report after an upgrade to INN 2.5.2 will probably contain incorrect statistics for incoming and outgoing articles because the beginning of the log files that will be used was generated by a previous version of INN.
A new version of innreport.conf is shipped with INN 2.5.2 but, in order to preserve any local changes, will not be automatically installed with make update. The changes are minor and not mandatory for the upgrade.
Changes from 2.5.1 to 2.5.2
* Julien Elie has implemented in innd the new version of the NNTP protocol described in RFC 3977, RFC 4643 and RFC 4644, and innd now recognizes the CAPABILITIES command. Despite these standards, three commands (IHAVE, CHECK and TAKETHIS) will continue, for interoperability reasons, to return a reject code (respectively 435, 438, and 439) when the command contains a syntax error instead of 501. The mandatory username argument for authenticated peers is not enforced in INN 2.5.2 but will be be enforced by INN 2.6.0 when it is released.
Major improvements are:
* innd now has a decent parser for NNTP commands. The parser is more correct (commands like "IHAVEZ<>", without a space between the command and its argument, are no longer valid) and allows leading and trailing whitespaces in commands. innd also now checks the length of the NNTP command sent by the client. If the command contains more than 512 bytes (or 497 bytes for an argument), an error is returned and the command is discarded. After ten unrecognized commands, innd closes the connection with the appropriate code (400 instead of 500).
* The output of the HELP command specifies the arguments expected by NNTP commands, similar to nnrpd's HELP command.
* LIST ACTIVE, LIST ACTIVE.TIMES and LIST NEWSGROUPS now allow an optional wildmat argument to restrict the results of those commands to specific newsgroups.
* When using HEAD or STAT with an article number or a range, 412 (no group selected) is now returned instead of 501 (syntax error).
* Jeffrey M. Vinocur has implemented support in both innd and nnrpd for whitespace in usernames/passwords provided with AUTHINFO USER/PASS. They were previously treated as invalid arguments or incorrectly parsed. innd and nnrpd now treat everything after the first whitespace character following AUTHINFO USER/PASS, up to, but not including, the final CRLF, as the username/password, in conformity with RFC 4643.
* The syntax of message-IDs is now based on RFC 5536 (USEFOR) instead of RFC 1036. The major change is that quoted-pairs have been removed from the syntax.
* The Perl and Python filters for innd now check the message-ID of articles arriving through TAKETHIS. Only CHECK and IHAVE commands previously used them.
* Case-insensitive matches are now used for distributions, path identities, IMAP commands, header names, and control commands. (Newsgroups are still matched case-sensitively.) Message-IDs are case-sensitively matched, except for history hashes.
* The new Archive:, Archive-At:, Comments:, and Summary: header fields defined in RFC 5064 and RFC 5536 can be used in innd filters. nnrpd now checks at injection time that an article does not contain an Injection-Info: header, that an Injection-Date: header (if provided) is valid, and that the Path: header does not contain ".POSTED". Note that INN does not yet generate these two injection fields or include the new Path: header field ".POSTED" keyword. These new features will be in the next major release of INN.
* LIST SUBSCRIPTIONS now accepts an optional wildmat argument to restrict the results of this command to specific newsgroups.
* nnrpd now supports a new LIST variant named COUNTS. LIST COUNTS is a combination of LIST ACTIVE and GROUP. It returns the same result as LIST ACTIVE except that the number of articles in a newsgroup is inserted before its status.
* A new flag has been added to newsfeeds entries: "Aj", when present, says to feed articles accepted and filed in "junk" (due to *wanttrash*) to peers based on their newsfeeds feed patterns applied to the Newsgroups: header as though the article were accepted and all those groups were locally carried. This is useful if you want to run INN with a minimal active file and propagate all posts. Thanks to Andrew Gierth for the patch.
* A new parameter has been added to inn.conf: *logtrash* defines whether a line for articles posted to groups not locally carried by the news server should be added in the news log file to report unwanted newsgroups. The default is true but it can be useful to set it to false (especially when *wanttrash* is also used).
* The procbatchdir keyword has been added to news.daily to specify the backlog directory of innfeed. This is useful when several instances of innfeed are running or when its configuration file is not the default one.
* sm now supports a new flag, -c, which shows a decoded form of the storage API token. This was previously done by the contrib showtoken script developed by Olaf Titz and Marco d'Itri.
* The O flag in newsfeeds now relies on the contents of the Injection-Info: header field if it is present to determine the origin of an article. It falls back on X-Trace: if there is no Injection-Info: header field.
* A new "unsigned long" type bas been added to the configuration parser. It will properly warn the news administrator when a variable supposed to be positive contains a negative integer. It will prevent INN from crashing due to misconfiguration at several places where it did not expect negative values.
* innxbatch and innxmit now recognize the new 403 code introduced by RFC 3977 for a problem preventing the requested action from being taken.
* HDR and OVER commands now return the correct 423 code (instead of 420) when the current article number is used but the article no longer exists.
* actsync, inews, innxbatch, innxmit, nntpget and rnews can now authenticate to news servers which only expect a username, without password, conforming to RFC 4643.
* The keyword generation code now generates a Keywords: header only if the original article does not already have one. The generated Keywords: header no longer begins with a comma. If keyword generation is set to true in inn.conf but the Keywords: header is not stored in the overview, the news administrator is warned and keyword generation deactivated, since it exists only to populate the overview data.
* Two segfaults in keyword generation were fixed. The first occurred when an article already had a Keywords: header longer than the *keylimit* parameter. The second was caused by a possible invalid pointer beyond the newly allocated Keywords: header.
* Fixed innd handling of empty lines. innd was not properly discarding an empty command and was closing the connection when it received only whitespace in a command.
* Fixed a bug in how innd responded to reader commands when readers were not allowed. A superfluous blank line was sent in its response.
* Fixed a bug in innd's response to TAKETHIS when authentication is required. Previously, 480 code was returned immediately without accepting the multi-line data block first, which broke synchronization in the NNTP protocol.
* Fixed a bug in recognizing the article terminator when empty articles were fed to innd via IHAVE or TAKETHIS, leading to treating subsequent NNTP commands as part of the article.
* When innd could not provide information for LIST ACTIVE.TIMES and LIST NEWSGROUPS, it was returning an invalid error message without a response code. The proper 503 answer code is now returned.
* When an unauthenticated user tried to post an article, nnrpd replied 440 (posting not allowed) instead of the correct 480 (authentication required) response if the user might be able to post after authentication. Thanks to Daniel Weber for the bug report.
* Fixed a bug in both innd and nnrpd answers to LIST commands where the output was not checked for valid dot stuffing.
* Fixed a bug leading to junked non-control articles being sent to control-only feeds, and also fixed handling of poisoned control groups. Thanks to Andrew Gierth for the patch.
* Fixed a bug in innreport leading to incorrect summing of innd stats when *hostname* was set to an IPv6 address instead of a fully-qualified domain name. Thanks to Petr Novopashenniy for the bug report.
* Changed how innreport uses innd and innfeed checkpoint messages. Previously, connections held open for multiple days led to skewed and incorrect statistics on how many articles had been received or sent. The count is now more accurate and, for each connection of a feed, only depends on *incominglogfrequency* in inn.conf and *stats-period* in innfeed.conf.
* Fixed a bug in nnrpd Perl filter: a header field whose name begins with the name of a standardized header field was not properly handled.
* Fixed a bug in how innd was parsing Message-ID: and Supersedes: headers which contained trailing whitespace. The article was corrupted by an unexpected "\r" in the middle of the header. nnrpd now checks the syntax of the Message-ID: header field, if present.
* Fixed various bugs in how leading whitespace was treated in headers. The HDR, XHDR and XPAT commands were not properly showing leading whitespace in header values. Lone "\n" and "\r" characters are now changed into spaces and "\r\n" is just removed. archive, makehistory, and tdx-util now keep leading whitespace in headers when generating overview data, and archive now changes "\n" (when not preceded by "\r") into a space when generating overview data.
* Fixed a bug in the generation of overview data which may corrupt previously generated overview data when a pseudo Xref: header field is injected in an extra overview field.
* Fixed a bug in the parsing of the *ovgrouppat* wildmat in inn.conf that prevented overview data from being generated when poisoned groups were specified but a latter sub-pattern matched the group. A uwildmat expression is now correctly handled, and a potential segfault has been fixed. Thanks to Dieter Stussy for the bug report.
* Fixed a bug when HDR, XHDR and XPAT were used when *virtualhost* was set to true in readers.conf. The Xref: header of articles posted to only one newsgroup appeared empty.
* Fixed a bug in tdx-util in parsing empty overview fields when called with -A or -F.
* Fixed a bug in cvtbatch, which was returning only the size of the headers of an article when the "b" parameter was used with the -w flag. It now correctly returns the size of the whole article, which is what "b" was documented to do. cvtbatch also has a new "t" parameter, which can be used with the -w flag to retrieve the arrival time of an article.
* Fixed a bug in how mailpost handles cross-posting feature. It was not properly detaching from sendmail. Thanks to Harald Dunkel for the patch.
* Fixed a bug in the newsfeeds C flag: the count of followup groups was one less than the real number. When the value of the Followup-To: header field is "poster", it is no longer considered to be a followup. Thanks to Dieter Stussy for the patch.
* When using tradindexed, the overview data for a cancelled article is now immediately removed from the overview. Thanks to Lars Magne Ingebrigtsen for the patch.
* batcher has not supported the retrieval of an article with its file name for a long time. The -S flag has therefore been removed.
* inews no longer rejects articles that contain more than 50 header fields. Thanks to Torsten Jerzembeck for the bug report.
* news.daily no longer sends superfluous mails when the nomail keyword is given. Mail is only sent when there is real output. Previously, there would always be headings and empty lines left over from the structuring of the full report, which are now ommitted. Also, the output of programs executed with postexec is now included in the regular mail. Thanks to Florian Schlichting for the patch.
* innconfval no longer maps NULL string or list values to an empty string or list and instead maps them to undefined values. This fixes an issue reported by Kamil Jonca: nnrpd was inserting an empty Organization: header when the *organization* parameter in inn.conf was unset.
* Other minor bug fixes and documentation improvements.
reset later on) and had a typo. Add a dependency on gnupg instead to make
sure a pgp (gpg) binary is around to be used (provision should be made to
be able to use eg netpgp instead .. to be done later, hopefully after
someone else created the 'pick my favourite pgp version' infrastructure :> ).
Major changes from 2.5.0 to 2.5.1
* Fixed a segfault in imap_connection which could occur when SASL was
used.
* Fixed a segfault in the keyword generation code which was assuming
that an article was nul-terminated. Fixed another segfault in the
keyword generation code when an article already contained a Keywords:
header. Thanks to Nix for the bug reports.
* Owing to the US-CERT vulnerability note VU#238019, Cyrus SASL library
has slightly changed. imap_connection and nnrpd now handle that
change. Otherwise, some answers are too long to be properly computed
during SASL exchanges.
* Fixed a memory allocation problem which caused nnrpd to die when
retrieving via HDR/XHDR/XPAT the contents of an extra overview field
absent from the headers of an article. The NEWNEWS command was also
affected on very rare cases. Thanks to Tim Woodall for the bug
report.
* HDR/XHDR/XPAT answers are now robust when the overview database is
inconsistent. When the overview schema was modified without the
overview database being rebuilt, wrong results could be returned for
extra fields (especially a random portion of some other header). The
desired header name is now explicitly searched for in the overview
information.
* Fixed the source which is logged to the news log file for local
postings when the local server is not listed in incoming.conf. A
wrong name was used, taken amongst known peers. The source is now
logged as "localhost".
* Fixed a bug in the timecaf storage method: only the first 65535
articles could be retrievable in a CAF, though everything was properly
stored. (A Crunched Article File contains all the articles that
arrive to the news server during 256 seconds.)
The storage token now uses 4 bytes to store the article sequence
number for timecaf, instead of only 2 bytes. Thanks to Kamil Jonca
for the bug report and also the patch.
* Fixed a bug in both timecaf and timehash which prevented them from
working on systems where short ints were not 16-bit integers.
* When there is not enough space to write an entire CAF header, the
timecaf storage manager now uses a larger blocksize. On 32-bit
systems, the CAF header is about 300 bytes, leaving about 200 bytes
for the free bitmap index (the remaining of a 512-byte blocksize). On
64-bit systems, the size of the CAF header could exceed 512 bytes,
thus leaving no room for the free bitmap index. A 1 KB blocksize is
then used, or a larger size if need be.
* A new CNFS version has been introduced by Miquel van Smoorenburg in
the CNFS header. CNFSv4 uses 4 KB blocks instead of 512 bytes, which
more particularly makes writes faster. CNFSv4 supports
files/partitions up to 16 TB with a 4 KB blocksize.
Existing CNFS buffers are kept unchanged; only new CNFS buffers are
initialized with that new version.
* grephistory -l now returns the contents of the expires history field
as well as the hash of the message-ID. Besides, when the storage API
token does not exist, grephistory -v now also returns the hash of the
requested message-ID.
* The check on cancel messages when *verifycancels* is set to true in
inn.conf has been changed to verify that at least one newsgroup in the
cancel message can be found in the article to be cancelled. This new
feature is from Christopher Biedl.
The previous behaviour was to check whether the cancel message is from
the same person as the original post, which is extremely easy to
spoof; besides, RFC 5537 (USEPRO) mentions that "cancel control
messages are not required to contain From: and Sender: header fields
matching the target message. This requirement only encouraged cancel
issuers to conceal their identity and provided no security".
* The way the "/remember/" line in expire.ctl works has changed.
History retention for an article was done according to its original
arrival time; it is now according to its original posting date.
Otherwise, unnecessary data may be kept too long in the history file.
To achieve that, the HISremember() function in history API now expects
a fourth parameter: the article posting time.
Note that article expiration has not changed and is still based on
arrival time, unless the -p flag is passed to expire or expireover, in
which case posting time is used.
* The default value for "/remember/" has changed from 10 to 11 because
it should be one more than the *artcutoff* parameter in inn.conf, so
that articles posted one day into the future are properly retained in
history.
* auth_krb5 has been rewritten by Russ Allbery to use modern Kerberos
APIs. Note that using ckpasswd with PAM support and a Kerberos PAM
module instead of this authenticator is still recommended.
* A new -L flag has been added by Jonathan Kamens to makehistory so as
to specify a load average limit. If the system load average exceeds
the specified limit, makehistory sleeps until it goes below the limit.
* As UTF-8 is the default character set in RFC 3977, "ctlinnd pause",
"ctlinnd readers", "ctlinnd reject", "ctlinnd reserve", "ctlinnd
throttle" and "nnrpd -r" commands now require the given reason to be
encoded in UTF-8, so that it can be properly sent to news readers.
The creator's name given to "ctlinnd newgroup" is also expected to be
encoded in UTF-8.
* The output of consistency checks for article storage and the history
file no longer appears by default when "cnfsstat -a" is used. A new
-v flag has been added to cnfsstat so as to see it.
* The default path for TLS certificates has changed from *pathnews*/lib
to *pathetc*. It only affects new INN installations or generations of
certificates with "make cert". Besides, a default value has been
added to *tlscapath* because it is required by nnrpd when TLS is used.
* gzip(1) is now the default UUCP batcher in send-uucp instead of
compress(1) because gzip is more widely available than compress, due
to old patent issues. Note that there is no impact on decompression
as it is handled by rnews.
* cnfsheadconf now uses the Perl core module "Math::BigInt" rather than
the deprecated bigint.pl library. When used without specifying a CNFS
buffer, it now properly displays the status of all CNFS buffers.
- Previous versions of INN had an optimization for handling XHDR
Newsgroups that used the Xref header from overview. While this does
make the command much faster, it doesn't produce accurate results and
breaks the NNTP protocol, so this optimization has been removed.
- Fixed a bug in innd that allowed it to accept articles with duplicated
headers if the header occurred an odd number of times. Modified the
programs for rebuilding overview to use the last Xref header if there
are multiple ones to avoid problems with spools that contain such
invalid articles.
- Fixed yet another problem with verifying that a user has permissions
to approve posts to a moderated group. Thanks, Jens Schlegel.
- Increase the send and receive buffer on the Unix domain socket used by
ctlinnd. This should allow longer replies (particularly for innstat)
on platforms with very low default Unix domain socket buffer sizes.
- rnews's handling of articles with nul characters, NNTP errors, header
problems, and deferrals has been significantly improved.
- Thomas Parmelan added support to send-uucp for specifying the funnel
or exploder site to flush for feeds managed through one and fixed a
problem with picking up old stranded work files.
- INN is now licensed under a less restrictive license (about as
minimally restrictive as possible shy of public domain), and the
clause similar to the old BSD advertising clause has been dropped.
- make install and make update now always install the newly built
binaries, rather than only installing them if the modification times
are newer. This is the behavior that people expect. make install now
also automatically builds a new (empty) history database if one
doesn't already exist.
- The embedded Tcl filter code has been disabled (and will be removed
entirely in the next major release of INN). It hasn't worked for some
time and causes innd crashes if compiled in (even if not used). If
someone wants to step forward and maintain it, I recommend starting
from scratch and emulating the Perl and Python filters.
- ctlinnd should now successfully handle messages from INN up to the
maximum allowable packet size in the protocol, fixing problems sites
with many active peers were having with innstat output.
- Overview generation has been fixed in both makehistory and innd to
follow the rules in the latest NNTP draft rather than just replacing
special characters with spaces. This means that the unfolding of
folded header lines will not introduce additional, incorrect
whitespace in the overview data.
- nnrpd now uniformly responds with a 480 or 502 status code to attempts
to read a newsgroup to which the user does not have access, depending
on whether the user has authenticated. Previously, it returned a 411
status code, claiming the group didn't exist, which confuses the
reactive authentication capability of news readers.
- If a user is not authorized to approve articles (using the A access
control in readers.conf), articles that include Approved headers will
be rejected even if posted to unmoderated groups. Some other site may
consider that group to be moderated.
- The configuration parser used for readers.conf and others now
correctly handles "#" inside quoted strings and is more robust against
unmatched double quotes.
- Messages mailed to moderators had two spaces after the colons in the
headers, rather than one. This bug has been fixed.
- A bug that could cause heap corruption and random crashes in innd if
INN were compiled with Python support has been fixed.
- Some problems with innd's tracking of article size and enforcement of
the configured maximum article size have been fixed.
- pgpverify will now correctly verify signatures generated by GnuPG and
better supports GnuPG as the PGP implementation.
- INN's code should now be more 64-bit clean in its handling of size_t,
pointer differences, and casting of pointers, correcting problems that
showed up on 64-bit platforms like AMD64.
- Improved the error reporting in the history database code, in inews,
in controlchan, and in expire.
- Many other more minor bug fixes, optimization improvements, and
documentation fixes.
- Clients using POST are no longer permitted to provide an Injector-Info
header.
- Fixed a bug causing posts with Followup-To set to a moderated group to
be rejected if the posting user didn't have permission to approve
postings.
- Fixed bugs in inncheck with setuid rnews or setgid inews, in
innconfval with inn.conf parameters containing shell metacharacters
but no spaces, and in parsedate.y with some versions of yacc. Fixed a
variety of size-related printf format warnings (e.g., %d vs. %ld)
thanks to the work of Winfried Szukalski.
While INN 2.3.5 is not the latest officially released version it is the
latest one for which an IPv6 patch is available.
- LIST ACTIVE no longer returns data when given a single group argument
if the client is not authorized to read that group.
- XHDR and XPAT weren't correctly parsing article headers, resulting in
searches for the header "newsgroup" matching the header "newsgroups".
- Made CNFS more robust against crashes by actually syncing the cycbuff
headers to disk as was originally intended. Fixed a memory leak in
the tradspool code.
- Two bugs in pgpverify when using GnuPG were fixed: it now correctly
checks for gpgv (rather than pgp) when told to use GnuPG and expects
the keyring to be pubring.gpg (not pubring.pgp).
- Substantial updates to the sample provided control.ctl file.
- Compilation fixes with Perl 5.8.0, BerkeleyDB 4.x, current versions of
Linux (including with large file support), and Tru64. inndf fixes for
ReiserFS.
- Various bugs in the header handling in nnrpd have been fixed,
including hangs when using virtual domains and improper processing of
folded headers under certain circumstances.
- Other minor bug fixes and documentation improvements.
While INN 2.3.4 is not the latest officially released version it is the
latest one for which an IPv6 patch is available.
- pgpverify now supports using GnuPG to check signatures (rather than
PGP) without the pgpgpg wrapper. GnuPG can check both old-style RSA
signatures and new OpenPGP signatures and is recommended over PGP 2.6.
If you have GnuPG installed, pgpverify will use it rather than PGP,
which means that you may have to create a new key ring for GnuPG to
use to verify signatures if you were previously using PGP.
- An item that was noted in NEWS below but was inadvertently left out of
nnrpd has actually been added this time: users can no longer post
articles containing Approved: headers to moderated groups by default;
they must be specifically given that permission with the access:
parameter in readers.conf. See the man page for more details.
- Two bugs in repacking overview index files and a reliability bug with
writing overview data were all fixed in the tradindexed overview
method, hopefully making it somewhat more reliable, particularly for
makehistory.
- If rc.news.local exists in the INN binary directory, it will be run
with the start or stop argument whenever rc.news is run. This is
available as a hook for local startup and shutdown code.
- The default history table hash sizes were increased because a
too-small value can cause serious performance problems (whereas a
too-large hash just wastes a bit of disk space).
- The sample control.ctl file has been extensively updated.
- Wildmat exclusions (@ and !) should now work properly in storage.conf
newsgroup patterns.
- The implementation of the -w flag for expireover was fixed;
previously, the value given to -w to change expireover's notion of the
current time was scaled by too much.
- Various other more minor bug fixes, standards compliance fixes, and
documentation improvements.
- innxmit can again handle regular filenames as input as well as storage
API tokens (allowing it to be used to import an old traditional
spool).
- Several problems with tagged-hash history files have been fixed thanks
to the debugging efforts of Andrew Gierth and Sang-yong Suh.
- A very long-standing (since INN 1.0!) NNTP protocol bug in nnrpd was
fixed. The response to an ARTICLE command retrieving a message by
message ID should have the message ID as the third word of the
response, not the fourth. Fixing this is reported to *possibly* cause
problems with some Netscape browsers, but other news servers correctly
follow the protocol.
- Some serious performance problems with expiration of tradspool should
now be at least somewhat alleviated. tradspool and timehash now know
how to output file names for removal rather than tokens, and fastrm's
ability to remove regular files has been restored. This should bring
expiration times for tradspool back to within a factor of two of
pre-storage-API expiration times.
- An item that was actually changed in 2.3.0 but wasn't noted in NEWS
when it should have been: Users can no longer post articles
containing Approved: headers to moderated groups by default; they must
be specifically given that permission with the access: parameter in
readers.conf. See the man page for more details.
- Added a sample subscriptions file and documentation for it and
innmail.
The changes required for this update were contributed by Bernd Ernesti
in PR pkg/13299.
