Changes in MySQL 5.7.33
Optimizer Notes
MySQL attempts to use an ordered index for any ORDER BY or GROUP BY query that has a LIMIT clause, overriding any other choices made by the optimizer, whenever it determines that this would result in faster execution. Because the algorithm for making this determination makes certain assumptions about data distribution and other conditions, it may not always be completely correct, and it is possible in some cases that choosing a different optimization for such queries can provide better performance. To handle such occurrences, it is now possible to disable this optimization by setting the optimizer_switch system variable's prefer_ordering_index flag to off.
For more information about this flag and examples of its use, see Switchable Optimizations, and LIMIT Query Optimization.
Our thanks to Jeremy Cole for the contribution.
Security Notes
The linked OpenSSL library for MySQL Server has been updated to version 1.1.1i. Issues fixed in the new OpenSSL version are described at https://www.openssl.org/news/cl111.txt and https://www.openssl.org/news/vulnerabilities.html.
Functionality Added or Changed
When invoked with the --all-databases option, mysqldump now dumps the mysql database first, so that when the dump file is reloaded, any accounts named in the DEFINER clause of other objects will already have been created.
Bugs Fixed
InnoDB: The full-text search synchronization thread attempted to read a previously-freed word from the index cache.
InnoDB: Calls to numa_all_nodes_ptr were replaced by the numa_get_mems_allowed() function. Thanks to Daniel Black for the contribution.
Replication: When the system variable transaction_write_set_extraction=XXHASH64 is set, which is the default in MySQL 8.0 and a requirement for Group Replication, the collection of writes for a transaction previously had no upper size limit. Now, for standard source to replica replication, the numeric limit on write sets specified by binlog_transaction_dependency_history_size is applied, after which the write set information is discarded but the transaction continues to execute. Because the write set information is then unavailable for the dependency calculation, the transaction is marked as non-concurrent, and is processed sequentially on the replica. For Group Replication, the process of extracting the writes from a transaction is required for conflict detection and certification on all group members, so the write set information cannot be discarded if the transaction is to complete. The byte limit set by group_replication_transaction_size_limit is applied instead of the numeric limit, and if the limit is exceeded, the transaction fails to execute.
Replication: As the number of replicas replicating from a semisynchronous source server increased, locking contention could result in a performance degradation. The locking mechanisms used by the plugins have been changed to use shared locks where possible, avoid unnecessary lock acquisitions, and limit callbacks. The new behaviors can be implemented by enabling the following system variables:
replication_sender_observe_commit_only=1 limits callbacks.
replication_optimize_for_static_plugin_config=1 adds shared locks and avoids unnecessary lock acquisitions. This system variable must be disabled if you want to uninstall the plugin.
Both system variables can be enabled before or after installing the semisynchronous replication plugin, and can be enabled while replication is running. Semisynchronous replication source servers can also get performance benefits from enabling these system variables, because they use the same locking mechanisms as the replicas.
Replication: On a multi-threaded replica where the commit order is preserved, worker threads must wait for all transactions that occur earlier in the relay log to commit before committing their own transactions. If a deadlock occurs because a thread waiting to commit a transaction later in the commit order has locked rows needed by a transaction earlier in the commit order, a deadlock detection algorithm signals the waiting thread to roll back its transaction. Previously, if transaction retries were not available, the worker thread that rolled back its transaction would exit immediately without signalling other worker threads in the commit order, which could stall replication. A worker thread in this situation now waits for its turn to call the rollback function, which means it signals the other threads correctly.
Replication: GTIDs are only available on a server instance up to the number of non-negative values for a signed 64-bit integer (2 to the power of 63 minus 1). If you set the value of gtid_purged to a number that approaches this limit, subsequent commits can cause the server to run out of GTIDs and take the action specified by binlog_error_action. From MySQL 8.0.23, a warning message is issued when the server instance is approaching the limit.
Microsoft Windows: On Windows, running the MySQL server as a service caused shared-memory connections to fail.
The server did not handle all cases of the WHERE_CONDITION optimization correctly.
For the engines which support primary key extension, when the total key length exceeded MAX_KEY_LENGTH or the number of key parts exceeded MAX_REF_PARTS, key parts of primary keys which did not fit within these limits were not added to the secondary key, but key parts of primary keys were unconditionally marked as part of secondary keys.
This led to a situation in which the secondary key was treated as a covering index, which meant sometimes the wrong access method was chosen.
This is fixed by modifying the way in which key parts of primary keys are added to secondary keys so that those which do not fit within which do not fit within the limits mentioned previously mentioned are cleared.
Privileges for some INFORMATION_SCHEMA tables were checked incorrectly.
In certain cases, the server did not handle multiply-nested subqueries correctly.
Certain accounts could cause server startup failure if the skip_name_resolve system variable was enabled.
Client programs could unexpectedly exit if communication packets contained bad data.
A buffer overflow in the client library was fixed.
mysql_config_editor incorrectly treated # in password values as a comment character.
Mostly, this is adapting the warning flags removed from the build and
those added to clang vs gcc.
Hoist removal of -Wno-stringop-truncation to always happen, because it
was separately on for clang and gcc, and it seems likely it would be
unknown on some other compiler.
- Fix: Revise the 08* test to use generic regexen on non-word directory
element separators to deal with MSW inconsistent use of them confusing
File::Spec::catpath.
- Added die, notes in placeholder lib.pm -- only visible when the
Makefile.PL is bypassed and Perl version check is skipped.
- Replace VERSION_FROM with VERSION in Makefile.PL to keep release info
consistent between Perl installation versions.
- Add ./version/v5.32.1
- Strict is unnecessary in 5.32.
- Test cleanups.
- Use canonpat on subdir and lib args to avoid including
- dir's with '//' in them.
*** Version 3.0.29 stable ***
- Security/Reliability:
- Fixed memory leaks when a response is buffered and the buffer
limit is reached or Privoxy is running out of memory.
Commits bbd53f1010b and 4490d451f9b. OVE-20201118-0001.
Sponsored by: Robert Klemme
- Fixed a memory leak in the show-status CGI handler when
no action files are configured. Commit c62254a686.
OVE-20201118-0002.
Sponsored by: Robert Klemme
- Fixed a memory leak in the show-status CGI handler when
no filter files are configured. Commit 1b1370f7a8a.
OVE-20201118-0003.
Sponsored by: Robert Klemme
- Fixes a memory leak when client tags are active.
Commit 245e1cf32. OVE-20201118-0004.
Sponsored by: Robert Klemme
- Fixed a memory leak if multiple filters are executed
and the last one is skipped due to a pcre error.
Commit 5cfb7bc8fe. OVE-20201118-0005.
- Prevent an unlikely dereference of a NULL-pointer that
could result in a crash if accept-intercepted-requests
was enabled, Privoxy failed to get the request destination
from the Host header and a memory allocation failed.
Commit 7530132349. CID 267165. OVE-20201118-0006.
- Fixed memory leaks in the client-tags CGI handler when
client tags are configured and memory allocations fail.
Commit cf5640eb2a. CID 267168. OVE-20201118-0007.
- Fixed memory leaks in the show-status CGI handler when memory
allocations fail. Commit 064eac5fd0 and commit fdee85c0bf3.
CID 305233. OVE-20201118-0008.
- General improvements:
- Added experimental https inspection support which allows to filter
https traffic. To enable it, install MbedTLS and configure with
--with-mbedtls, or install OpenSSL or LibreSSL and configure
with --with-openssl.
Afterwards configure the directives in section 7 of the
config file and enable the +https-inspection action.
Initial MbedTLS-based code contributed by Vaclav Svec,
initial OpenSSL support contributed by Maxim Antonov.
With help from Nedzad Hrnjica and Ho+ Ho+ Ho+.
Integration and improvements sponsored by Robert Klemme.
- pcrs: Request JIT compilation if it's supported and
the filter isn't dynamic. This can speed up filtering.
- Added support for Brotli decompression.
Sponsored by: Robert Klemme
- Added FEATURE_EXTENDED_STATISTICS to gather statistics for
block reasons and filter executions. To enable it, configure
with --enable-extended-statistics and visit
http://config.privoxy.org/show-status.
Sponsored by: Robert Klemme
- Use the IP_FREEBIND socket option, if defined. This allows
Privoxy to bind to not-yet assigned IP addresses which is
useful in failover environments.
Patch by Sam Varshavchik.
- Allow to use extended host patterns and vanilla host patterns
at the same time by prefixing extended host patterns with
"PCRE-HOST-PATTERN:". To enable this, configure with
--enable-pcre-host-patterns.
Sponsored by: Robert Klemme
- Added "Cross-origin resource sharing" (CORS) support.
This allows to access Privoxy's CGI interface via JavaScript from
another domain (white-listed with the new cors-allowed-origin directive).
Based on a patch by Nedzad Hrnjica.
Sponsored by: Robert Klemme.
- Add SOCKS5 username/password support.
Based on a patch by Sam, improved by Ivan Romanov.
Closes Patch#141 and solves TODO#105.
- Bump the maximum number of action and filter files
to 100 each.
Sponsored by: Robert Klemme
- Fixed handling of filters with "split-large-forms 1"
when using the CGI editor.
Reported by withoutname in #921.
- Better detect a mismatch of connection details when
figuring out whether or not a connection can be reused.
- Don't send a "Connection failure" message instead of the
"DNS failure" message.
Sponsored by: Robert Klemme
- Let LOG_LEVEL_REQUEST log all requests. Previously unencrypted
requests were only logged with LOG_LEVEL_REQUEST when they weren't
crunched (in which case they were logged with LOG_LEVEL_CRUNCH).
This was documented behaviour, but logging all requests seems more useful.
- Fixed locking around localtime() and gmtime().
- Removed OS/2 support. We haven't provided OS/2 packages in years,
it complicated the code and it depended on a fallback snprintf()
implementation which is GPLv2 only.
- Remove the fallback snprintf() implementation
Now that OS/2 support is gone we no longer need it.
- Fixed a bunch of format specifiers log messages.
- Added a missing apostrophe in the 'More Privoxy' menu.
- Explicitly prevent use of FEATURE_CONNECTION_SHARING
without FEATURE_CONNECTION_KEEP_ALIVE. It makes no sense
and does not compile anyway.
Sponsored by: Robert Klemme
- Fix build without FEATURE_CONNECTION_KEEP_ALIVE.
Sponsored by: Robert Klemme
- Downgrade the 'Graceful termination requested' message
to LOG_LEVEL_INFO as it isn't an error.
Sponsored by: Robert Klemme
- decompress_iob(): Downgrade the no-content message to LOG_LEVEL_RE_FILTER
While at it, fix a typo in a comment.
Sponsored by: Robert Klemme
- Fixed a couple of cppcheck warnings.
- Rename LOG_LEVEL_GPC to LOG_LEVEL_REQUEST.
Only the shadow knows what "GPC" is supposed to stand for.
- Remove SourceForge references in copyright headers.
- Upgrade a bunch of links to the homepage to https://.
- Add 'no-brotli-accepted' filter which prevents the
use of Brotli compression.
- Changed license for pcrs to GPLv2+ after getting the
permission from Andreas. This allows to redistribute
Privoxy under the GPLv3 which is required when linking
to future mbedTLS versions which are expected to be
licensed under the Apache 2.0 license only.
- Updated a bunch of tests that have to expect status code 403
now after r1.168/070e904afa5.
- Lowercase the host name in the request line.
- Only set SOURCE_DATE_EPOCH if it's not already set so
distributions can overwrite it through the environment.
- Documentation changes:
- Explain that Privoxy has to be distributed under the
GPLv3 (or later) when linked with an MbedTLS version
that is licensed under the Apache 2.0 license.
- Import the GNU GPLv3 and include it the user manual.
- Clarify FEATURE_FORCE_LOAD's description. It allows to bypass
blocking not filtering and only does it if blocks aren't enforced.
Reported by: Robert Klemme
- FAQ: Remove Zwiebelfreunde e.V. from the list of fiduciary sponsors
As of 2021 they no longer handle donations for foreign organisations
due to lack of resources.
- FAQ: Remove an obsolete comment with a link to the long-gone PDF manual.
- FAQ: Add a link to the TODO list.
- FAQ: Change the sponsor amounts to USD slightly rounding the
converted amounts up to get simple numbers.
Receiving USD is apparently easier for SPI and SPI is
preferred by sponsors as they can send invoices.
- Advertise the client-tags CGI page in the user manual.
- Stop advertising the show-version CGI page which no longer exists.
- Add yet another reason why +prevent-compression may cause problems.
- Don't claim that contributors need ssh. It's only needed for committers.
- Replace obsolete CVS instructions with Git instructions.
- Remove an obsolete comment
- Config file changes:
- Change the suggested default-server-timeout to 5 to match the
suggested keep-alive-timeout. Otherwise using the defaults would
result in Privoxy reducing the default-server-timeout and logging
an error message.
Sponsored by: Robert Klemme
- Update the 'debug 1' description.
- Add a missing 'client-specific-tag' directive.
- Comment out trusted-cgi-referer pointing to example.org.
- Action file improvements:
- Block requests to /(.*/)?piwik\.php
- Block requests to .connectaserver.de/
- Block requests to pixel.inforsea.com/
- Block requests to t.vi-serve.com/
- Block requests to .ioam.de/
- Block requests to t.9gag.com/img.gif
- Block requests to .pixel.parsely.com/ as image
- Block requests to pixel.wp.com/
- Disable fast-redirects for .librarything.com/
- Disable fast-redirects for issue.freebsdfoundation.org/
- Disable fast-redirects for .twitter.com/.*origin=http
- Unblock belco24.de/
- Add fast-redirects exception for .wikipedia.org/
- Add fast-redirects exception for oss-fuzz.com/
- Disable fast-redirects for .consensu.org/delivery/pixel\.php
and block the requests as image instead
- Unblock .adbinstaller.com/
Reported by lvm in #942.
- Unblock .adbshell.com
Reported by lvm in #942.
- Unblock .tagesschau.de/
- Disable fast-redirects for collector.githubapp.com/
and block requests to it as image instead
- Unblock 'ada*.'
- Add fast-redirects{} exception for sourcepoint.vice.com/
- Unblock adaway.org/
Reported by DRS David Soft in AF#945.
- Change two block reasons that previously were the same.
Sponsored by: Robert Klemme
- Added a +delay-response{} test.
- Updated the location of the development version
of default.action.master.
- Privoxy-Log-Parser:
- Added a --keep-date option to keep the date in highlighted messages.
- Highlight new log messages.
- Make gather_loglevel_clf_stats() more tolerant. While at it,
count all CLF messages as requests, even if the request is invalid.
- Only show HTTP version distribution if at least one version has been detected.
- Only show crunch statistics if crunches were detected.
- Warn if the request counts differ.
- Generate statistics if the log only contains LOG_LEVEL_CLF messages
so it can be used with vanilla webserver logs.
Previously Privoxy-specific "Request:" messages were required.
- Align the client-HTTP-version distribution like other distributions
- Bump version to 0.9.1
- Include status code distribution in the stats.
- Let the statistics include the size of the content Privoxy
transferred excluding HTTP headers.
- Get with the program and expect all requests to be logged with LOG_LEVEL_REQUEST.
It's no longer necessary to count both LOG_LEVEL_REQUEST and
LOG_LEVEL_CRUNCH messages to get the total number of requests.
- Leverage the LOG_LEVEL_CLF message to gather statistics that where
previously taken from LOG_LEVEL_HEADER lines. This results in less
confusing results if https inspection is enabled in which case there
are two LOG_LEVEL_HEADER lines with request lines.
Sponsored by: Robert Klemme
- Properly highlight the filter results message. Previously a brace got lost.
- Prefer the number of CLF lines to get the total number of requests
as it works with older Privoxy versions as well.
- Privoxy-Regression-Test:
- Turn curl's globbing mode off so we can allow more characters in URLs.
- Allow '[' and ']' in URLs.
- Include the action file when complaining about missing Sticky Actions.
- Fix a sentence in the documentation.
- Bump version to 0.7.1
- url-pattern-translator:
- Detect a couple of pattern prefixes case-insensitively.
Sponsored by: Robert Klemme
- Skip CLIENT-TAG patterns.
Sponsored by: Robert Klemme
- Skip patterns that have already been converted.
It should now be safe to "convert" a file multiple times.
Sponsored by: Robert Klemme
- Add the new 'PCRE-HOST-PATTERN:' prefix.
Sponsored by: Robert Klemme
pkgsrc changes:
- drop workaround for upstream bug fixed in this release
- rename patchfile per pkglint
- Changes for xlog version 2.0.20 - 2021-jan-30
* Updated cty.dat 20210127 (cty-3102)
* Added FST4 support and 5m/8m bands for ADIF 3.1.1
* Added /dev/ttyUSB0 and /dev/ttyUSB1 to the hamlib device menu
* Logeditor can hide TX(RST) and RX (RST) (useful for VHF contests)
* ADIF exports Distance and Azimuth correctly if enabled in Logeditor
* Imports ADIF from wsjt-x on UDP port 2333
* Fixed use-after-free error discovered by Chris K2CR and valgrind
* Fixed hash-table != null assert in dxcc.c
* Fixed GLib-GObject-WARNING in log.c
* Compiles cleanly with hamlib4 on ubuntu 18.04 and 20.04
platform tuple doesn't match the one skalibs was built with. In pkgsrc,
this almost certainly means the OS has been updated, and almost
certainly doesn't need to break anyone's update builds. Explicitly pass
the contents of ${PREFIX}/lib/skalibs/sysdeps/target as the --target of
those configure scripts, then make sure we don't cross-compile.
skalibs-using packages not needing this workaround can define
SKALIBS_TOLERATE_TARGET_SKEW=no.
Changes:
10.0.10:
Not found.
10.0.9:
The full changelog since Desktop and Android Tor Browser 10.0.8 is:
All Platforms
Update NoScript to 11.1.9
Windows + OS X + Linux
Update Firefox to 78.7.0esr
Bug 40249: Remove EOY 2020 Campaign
Build System
All Platforms
Update Go to 1.14.14
10.0.8:
The full changelog since Desktop and Android Tor Browser 10.0.7 is:
All Platforms
Update NoScript to 11.1.7
Windows + OS X + Linux
Update Firefox to 78.6.1esr
New --bool option to specify Boolean search query patterns (with
Google search syntax or fzf-like when used with -F to search strings
instead of regex patterns); new --and and --not options; new --dotall
option; updated --format to support -v; other improvements. More
coming soon!
NEW IN WAF 2.0.22
-----------------
* Fix stdin propagation with faulty vcvarsall scripts #2315
* Enable mixing Unix-style paths with destdir on Windows platforms #2337
* Fix shell escaping unit test parameters #2314
* Improve extras/clang_compilation_database and extras/swig compatibility #2336
* Propagate C++ flags to the Cuda compiler in extras/cuda #2311
* Fix detection of Qt 5.0.0 (preparation for Qt6) #2331
* Enable Haxe processing #2308
* Fix regression in MACOSX_DEPLOYMENT_TARGET caused by distutils #2330
* Fix extras/wafcache concurrent trimming issues #2312
* Fix extras/wafcache symlink handling #2327
6.1.1 - 2021-01-31
This patch updates our automatic code formatting to use shed, which includes autoflake, black, isort, and pyupgrade (issue #2780).
6.1.0 - 2021-01-29
This release teaches Hypothesis to distinguish between errors based on the __cause__ or __context__ of otherwise identical exceptions, which is particularly useful when internal errors can be wrapped by a library-specific or semantically appropriate exception such as:
try:
do_the_thing(foo, timeout=10)
except Exception as err:
raise FooError("Failed to do the thing") from err
Earlier versions of Hypothesis only see the FooError, while we can now distinguish a FooError raised because of e.g. an internal assertion from one raised because of a TimeoutExceeded exception.
6.0.4 - 2021-01-27
This release prevents a race condition inside recursive() strategies. The race condition occurs when the same recursive() strategy is shared among tests that are running in multiple threads (issue #2717).
Version 9.0.5 (2021-01-24)
--------------------------
.Bug fixes
- Use config newline setting in system attribute evaulation (thanks @hoadlck)
.Testing
- Update to deadsnakes/python@v2.0.2
ChangeLog:
Version 0.8.0, 2021-01-31
Recognise the volume, source, and section metadata. These are currently
only used by -Tman.
Convert all internal functions to return an error code on memory allocation
failure. Prior to this, these functions had a chance of exiting and
printing failure to stderr. Now, this is left as the responsibility of
the front-end. There's no significant API change except that all renderers
return a value.
Fix the difference engine in several subtle ways, improving the produced
scripts, and also fix crashes where similar text would match multiple
parts of the parse tree, resulting in assertions.
Re-write the -Tms and -Tman generator to use a completely different
internal algorithm. This algorithm, instead of formatting directly into
output, converts the AST into an array of output blocks marked either as
text, literal, macro, or font/colour change. An assembler for this array
manages newlines and spacing between blocks. This fixes all known instances
of unexpected line breaks and allows for significantly simplified handling
of text interspersed with macros (e.g., links, etc.). An API result of
this is that the tree passed to lowdown_nroff_rndr(3) is now const.
Recognise non-block and block lists for -Tlatex output.
Emit a UTF-8 preconv header to all -Tms and -Tman so that -Kutf8 need not
be passed to the formatter.
Remove the --nroff-hardwrap option, which needlessly complicates code
without benefit.
20.9 - 2021-01-29
~~~~~~~~~~~~~~~~~
* Run [isort](https://pypi.org/project/isort/) over the code base (:issue:`377`)
* Add support for the ``macosx_10_*_universal2`` platform tags (:issue:`379`)
* Introduce ``packaging.utils.parse_wheel_filename()`` and ``parse_sdist_filename()``
(:issue:`387` and :issue:`389`)
