Redis 6.0.4
===========
Upgrade urgency CRITICAL: this release fixes a severe replication bug.
Redis 6.0.4 fixes a critical replication bug caused by a new feature introduced
in Redis 6. The feature, called "meaningful offset" and strongly wanted by
myself (antirez) was an improvement that avoided that masters were no longer
able, during a failover where they were demoted to replicas, to partially
synchronize with the new master. In short the feature was able to avoid full
synchronizations with RDB. How did it work? By trimming the replication backlog
of the final "PING" commands the master was sending in the replication channel:
this way the replication offset would no longer go "after" the one of the
promoted replica, allowing the master to just continue in the same replication
history, receiving only a small data difference.
However after the introduction of the feature we (the Redis core team) quickly
understood there was something wrong: the apparently harmless feature had
many bugs, and the last bug we discovered, after a joined effort of multiple
people, we were not even able to fully understand after fixing it. Enough was
enough, we decided that the complexity cost of this feature was too high.
So Redis 6.0.4 removes the feature entirely, and fixes the data corruption that
it was able to cause.
However there are two facts to take in mind.
Fact 1: Setups using chained replication, that means that certain replicas
are replicating from other replicas, up to Redis 6.0.3 can experience data
corruption. For chained replication we mean that:
+--------+ +---------+ +-------------+
| master |--------->| replica |-------->| sub-replica |
+--------+ +---------+ +-------------+
People using chained replication SHOULD UPGRADE ASAP away from Redis 6.0.0,
6.0.1, 6.0.2 or 6.0.3 to Redis 6.0.4.
To be clear, people NOT using this setup, but having just replicas attached
directly to the master, SHOUDL NOT BE in danger of any problem. But we
are no longer confident on 6.0.x replication implementation complexities
so we suggest to upgrade to 6.0.4 to everybody using an older 6.0.3 release.
We just so far didn't find any bug that affects Redis 6.0.3 that does not
involve chained replication.
People starting with Redis 6.0.4 are fine. People with Redis 5 are fine.
People upgrading from Redis 5 to Redis 6.0.4 are fine.
TLDR: The problem is with users of 6.0.0, 6.0.1, 6.0.2, 6.0.3.
Fact 2: Upgrading from Redis 6.0.x to Redis 6.0.4, IF AND ONLY IF you
use chained replication, requires some extra care:
1. Once you attach your new Redis 6.0.4 instance as a replica of the current
Redis 6.0.x master, you should wait for the first full synchronization,
then you should promote it right away, if your setup involves chained
replication. Don't give it the time to do a new partial synchronization
in the case the link between the master and the replica will break in
the mean time.
2. As an additional care, you may want to set the replication ping period
to a very large value (for instance 1000000) using the following command:
CONFIG SET repl-ping-replica-period 1000000
Note that if you do "1" with care, "2" is not needed.
However if you do it, make sure to later restore it to its default:
CONFIG SET repl-ping-replica-period 10
So this is the main change in Redis 6. Later we'll find a different way in
order to achieve what we wanted to achieve with the Meaningful Offset feature,
but without the same complexity.
Other changes in this release:
* PSYNC2 tests improved.
* Fix a rare active defrag edge case bug leading to stagnation
* Fix Redis 6 asserting at startup in 32 bit systems.
* Redis 6 32 bit is now added back to our testing environments.
* Fix server crash for STRALGO command,
* Implement sendfile for RDB transfer.
* TLS fixes.
* Make replication more resistant by disconnecting the master if we
detect a protocol error. Basically we no longer accept inline protocol
from the master.
* Other improvements in the tests.
Changes in version 3.32.1:
Fix two long-standing bugs that allow malicious SQL statements to crash the process that is running SQLite. These bugs were announced by a third-party approximately 24 hours after the 3.32.0 release but are not specific to the 3.32.0 release.
Other minor compiler-warning fixes and whatnot.
PyPika is a Python API for building SQL queries. The motivation behind PyPika
is to provide a simple interface for building SQL queries without limiting the
flexibility of handwritten SQL. Designed with data analysis in mind, PyPika
leverages the builder design pattern to construct queries to avoid messy string
formatting and concatenation. It is also easily extended to take full advantage
of specific features of SQL database vendors.
Tortoise ORM is an easy-to-use asyncio ORM (Object Relational Mapper) inspired
by Django.
Tortoise ORM was build with relations in mind and admiration for the excellent
and popular Django ORM. It's engraved in it's design that you are working not
with just tables, you work with relational data.
SQLite Release 3.32.0:
Added support for approximate ANALYZE using the PRAGMA analysis_limit command.
Added the bytecode virtual table.
Add the checksum VFS shim to the set of run-time loadable extensions included in the source tree.
Added the iif() SQL function.
INSERT and UPDATE statements now always apply column affinity before computing CHECK constraints. This bug fix could, in theory, cause problems for legacy databases with unorthodox CHECK constraints the require the input type for an INSERT is different from the declared column type. See ticket 86ba67afafded936 for more information.
Added the sqlite3_create_filename(), sqlite3_free_filename(), and sqlite3_database_file_object() interfaces to better support of VFS shim implementations.
Increase the default upper bound on the number of parameters from 999 to 32766.
Added code for the UINT collating sequence as an optional loadable extension.
Enhancements to the CLI:
Add options to the .import command: --csv, --ascii, --skip
The .dump command now accepts multiple LIKE-pattern arguments and outputs the union of all matching tables.
Add the .oom command in debugging builds
Add the --bom option to the .excel, .output, and .once commands.
Enhance the .filectrl command to support the --schema option.
The UINT collating sequence extension is automatically loaded
The ESCAPE clause of a LIKE operator now overrides wildcard characters, so that the behavior now matches what PostgreSQL does.
Update ruby-activeldap to 6.0.2.
pkgsrc change: remove RUBY_VERSIONS_INCOMPATIBLE for ruby24.
6.0.2: 2020-05-19 {#release-6-0-2}
Improvements
* Added options to {ActiveLdap::Persistance#reload}.
[GitHub#176][Reported by Kevin McCormack]
* jndi: Improved DN escaping. [GitHub#178][Patch by Kevin McCormack]
Thanks
* Kevin McCormack
5.1.0
repmgr: remove BDR 2.x support
repmgr: don't query upstream's data directory
repmgr: rename --recovery-conf-only to --replication-conf-only
repmgr: ensure postgresql.auto.conf is created with correct permissions
repmgr: minimize requirement to check upstream data directory location
during "standby clone"
repmgr: warn about missing pg_rewind prerequisites when excuting
"standby clone"
repmgr: add --upstream option to "node check"
repmgr: report error code on follow/rejoin failure due to non-available
replication slot
repmgr: ensure "node rejoin" checks for available replication slots
repmgr: improve "standby switchover" completion checks
repmgr: add replication configuration file ownership check to
"standby switchover"
repmgr: check the demotion candidate's registered repmgr.conf file can
be found
repmgr: consolidate replication connection code
repmgr: check permissions for "pg_promote()" and fall back to pg_ctl
if necessary
repmgr: in --dry-run mode, display promote command which will be used
repmgr: enable "service_promote_command" in PostgreSQL 12
repmgr: accept option -S/--superuser for "node check"
Redis 6.0.3:
Upgrade urgency CRITICAL: a crash introduced in 6.0.2 is now fixed.
Redis 6.0.2:
Upgrade urgency MODERATE: many not critical bugfixes in different areas.
Critical fix to client side caching when
keys are evicted from the tracking table but
no notifications are sent.
The following are the most serious fix:
* XPENDING should not update consumer's seen-time
* optimize memory usage of deferred replies - fixed
* Fix CRC64 initialization outside the Redis server itself.
* stringmatchlen() should not expect null terminated strings.
* Cluster nodes availability checks improved when there is
high Pub/Sub load on the cluster bus.
* Redis Benchmark: Fix coredump because of double free
* Tracking: send eviction messages when evicting entries.
* rax.c updated from upstream antirez/rax.
* fix redis 6.0 not freeing closed connections during loading.
New features:
dd
* Support setcpuaffinity on linux/bsd
* Client Side Caching: Add Tracking Prefix Number Stats in Server Info
* Add --user argument to redis-benchmark.c (ACL)
installed, anyway, resulting in a PLIST error.
ldb's snazzy little build system has no official configuration option
for this, so we reach in and flick the switch by hand.
0.36.5:
- Added support for dictionary input in CompositeType
- Added new EnrichedDateTime and EnrichedDate types
- Using String instead of LargeBinary for impl of EncryptedType
- Added support for JSONType in EncryptedType
0.36.4:
- Added jsonb_sql function
- Drop py27 support
1.3.17
orm
[orm] [usecase]
Added an accessor Comparator.expressions which provides access to the group of columns mapped under a multi-column ColumnProperty attribute.
[orm] [usecase]
Introduce relationship.sync_backref flag in a relationship to control if the synchronization events that mutate the in-Python attributes are added. This supersedes the previous change 5149, which warned that viewonly=True relationship target of a back_populates or backref configuration would be disallowed.
[orm] [bug]
Fixed bug where using with_polymorphic() as the target of a join via RelationshipComparator.of_type() on a mapper that already has a subquery-based with_polymorphic setting that’s equivalent to the one requested would not correctly alias the ON clause in the join.
[orm] [bug]
Fixed issue in the area of where loader options such as selectinload() interact with the baked query system, such that the caching of a query is not supposed to occur if the loader options themselves have elements such as with_polymorphic() objects in them that currently are not cache-compatible. The baked loader could sometimes not fully invalidate itself in these some of these scenarios leading to missed eager loads.
[orm] [bug]
Modified the internal “identity set” implementation, which is a set that hashes objects on their id() rather than their hash values, to not actually call the __hash__() method of the objects, which are typically user-mapped objects. Some methods were calling this method as a side effect of the implementation.
[orm] [bug]
An informative error message is raised when an ORM many-to-one comparison is attempted against an object that is not an actual mapped instance. Comparisons such as those to scalar subqueries aren’t supported; generalized comparison with subqueries is better achieved using Comparator.has().
engine
[engine] [bug]
Fixed fairly critical issue where the DBAPI connection could be returned to the connection pool while still in an un-rolled-back state. The reset agent responsible for rolling back the connection could be corrupted in the case that the transaction was “closed” without being rolled back or committed, which can occur in some scenarios when using ORM sessions and emitting .close() in a certain pattern involving savepoints. The fix ensures that the reset agent is always active.
schema
[schema] [bug]
Fixed issue where an Index that is deferred in being associated with a table, such as as when it contains a Column that is not associated with any Table yet, would fail to attach correctly if it also contained a non table-oriented expession.
[schema] [bug]
A warning is emitted when making use of the MetaData.sorted_tables attribute as well as the sort_tables() function, and the given tables cannot be correctly sorted due to a cyclic dependency between foreign key constraints. In this case, the functions will no longer sort the involved tables by foreign key, and a warning will be emitted. Other tables that are not part of the cycle will still be returned in dependency order. Previously, the sorted_table routines would return a collection that would unconditionally omit all foreign keys when a cycle was detected, and no warning was emitted.
[schema]
Add comment attribute to Column __repr__ method.
postgresql
[postgresql] [usecase]
Added support for columns or type ARRAY of Enum, JSON or JSONB in PostgreSQL. Previously a workaround was required in these use cases.
[postgresql] [usecase]
Raise an explicit CompileError when adding a table with a column of type ARRAY of Enum configured with Enum.native_enum set to False when Enum.create_constraint is not set to False
mssql
[mssql] [bug] [reflection]
Fix a regression introduced by the reflection of computed column in MSSQL when using the legacy TDS version 4.2. The dialect will try to detect the protocol version of first connect and run in compatibility mode if it cannot detect it.
[mssql] [bug] [reflection]
Fix a regression introduced by the reflection of computed column in MSSQL when using SQL server versions before 2012, which does not support the concat function.
oracle
[oracle] [bug]
Some modifications to how the cx_oracle dialect sets up per-column outputtype handlers for LOB and numeric datatypes to adjust for potential changes coming in cx_Oracle 8.
[oracle] [bug] [performance]
Changed the implementation of fetching CLOB and BLOB objects to use cx_Oracle’s native implementation which fetches CLOB/BLOB objects inline with other result columns, rather than performing a separate fetch. As always, this can be disabled by setting auto_convert_lobs to False.
As part of this change, the behavior of a CLOB that was given a blank string on INSERT now returns None on SELECT, which is now consistent with that of VARCHAR on Oracle.
firebird
[firebird] [change]
Adjusted dialect loading for firebird:// URIs so the external sqlalchemy-firebird dialect will be used if it has been installed, otherwise fall back to the (now deprecated) internal Firebird dialect.
