Please consult the Solr Upgrade notes for details and links specified below:
https://github.com/apache/solr/blob/main/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
====================================================================
Changelog:
Solr 8.11
See the 8.11 Release Notes for an overview of the main new features of Solr 8.11.
When upgrading to 8.11.x users should be aware of the following major changes from 8.10.
Support for Multiple Authentication Schemes
Two new authentication and authorization plugins provide support for configuring multiple authentication schemes.
The MultiAuthPlugin allows combining two or more authentication approaches, such as JWT and Basic authentication.
The MultiAuthRuleBasedAuthorizationPlugin is used when the MultiAuthPlugin is also in use, and combines the various roles defined for all plugins to determine the proper role assignment for the user account.
For information on configuring these plugins, see the following sections:
Combining Basic Authentication with Other Schemes
Multiple Authorization Plugins
ZooKeeper chroot
It’s now possible to create the ZooKeeper chroot at startup if it does not already exist. See the section Using the -z Parameter with bin/solr for an example.
Other Changes
A few other minor changes are worth noting:
The config-read pre-defined permission now correctly governs access for various configuration-related APIs. See also Predefined Permissions.
The S3BackupRepository supports configuring the AWS Profile, if necessary. See also S3BackupRepository.
Additionally, backups will now properly succeed after SPLITSHARD operations, and will correctly handle incremental backup purges.
SolrJ now supports uploading configsets.
Solr 8.10
See the 8.10 Release Notes for an overview of the main new features of Solr 8.10.
When upgrading to 8.10.x users should be aware of the following major changes from 8.9.
Schema Designer UI
A new screen has been added to the Admin UI that allows you to interactively design a Solr schema using your documents.
The designer screen provides a safe environment for you to:
Upload or paste sample documents to identify fields.
Get a "first" guess at what Solr thinks the field types in the fields should be.
Edit fields, field types, dynamic fields, and supporting files.
See how a field’s analysis will impact your text.
Test how schema changes will impact query-time behavior.
Save your changes to a configset to use with a new collection.
See the section Schema Designer for full details.
Backups in S3
Following the redesign of backups in Solr 8.8 that allowed storage of incremental backups in Google Cloud environments, Solr 8.10 provides support for storing backups in Amazon S3 buckets.
See the section S3BackupRepository for how to configure.
Security Admin UI
Solr’s Admin UI also got a new screen to support management of users, roles, and permissions.
The new UI works when authentication and/or authorization has been enabled with bin/solr auth or by manually installing a security.json file. Before this, it provides a warning that your Solr instance is unsecured.
See the section Security UI for details.
Solr SQL Improvements
A number of improvements have been made in Solr’s SQL functionality:
Support added for LIKE, IS NOT NULL, IS NULL, and wildcards (for simplistic LIKE functionality).
Two new aggregation functions, COUNT(DISTINCT field) and APPROX_COUNT_DISTINCT(field), have been added.
Queries using an ORDER BY clause can support OFFSET and FETCH operations.
Multi-valued fields can now be returned.
User permissions have been simplified so access to query endpoints /sql, /select, and /export is sufficient for full access for all SQL queries.
shards.preference
A new option for the shards.preference parameter allows selection of nodes based on whether or not the replica is a leader. Now adding shards.preference=replica.leader:false will limit queries only to replicas which are not currently their shard’s leader.
See the section shards.preference Parameter for details and examples.
Metrics & Prometheus Exporter
A new expr option in the Metrics API allows for more advanced filtering of metrics based on regular expressions. See the section Metrics API for examples.
The Prometheus Exporter’s default solr-exporter.config has been improved to use the new expr option in the Metrics API to get a smaller set of metrics. The default metrics exported still include most metrics, but the configuration will be easier to trim as needed. This should help provide performance improvements in busy clusters being monitored by Prometheus.
ZooKeeper Credentials
ZooKeeper credentials can now be stored in a file whose location is defined with a system property instead of being passed in plain-text. See Out of the Box Credential Implementations for how to set this up.
Solr 8.9
See the 8.9 Release Notes for an overview of the main new features of Solr 8.9.
When upgrading to 8.9.x users should be aware of the following major changes from 8.8.
Backup and Restore
Solr 8.9 introduces extensive changes to Solr’s backup and restore support.
A new backup format has been introduced in Solr 8.9 which replaces the previous snapshot-based backup. This new format enables ‘incremental’ backups. Repeated backups to a given location will take advantage of the data stored by their predecessors and will only operate on files that have changed since the previous backup. This is supported by default, simply by storing each backup file in the same location.
The old and new formats are not compatible, although backups in the old format, a full snapshot of all files, can still be used to restore to Solr for the time-being. The old format is officially deprecated, and support for it is likely to be removed in Solr 9.0.
For the time-being the old format can be created by defining a parameter incremental=false. Again, though, this support is likely to be removed in Solr 9.0.
More documentation on backups is available at Backup and Restore.
New Collections API commands for backups:
LISTBACKUP: Lists information about each backup stored at the specified repository location. See List Backups for more details.
DELETEBACKUP: Deletes specified backups from the repository. See Delete Backups for more details.
A new option for backup repository is also available in 8.9, which is to use Google Cloud Storage (GCS). This is a contrib (located in contrib/gcs-repository). See GCSBackupRepository for configuration details. The Solr community is working to add support for S3 buckets in the near future.
Nested Docs
Child Doc Transformer’s childFilter parameter no longer applies query syntax escaping because it’s inconsistent with the rest of Solr and was limiting. This refers to [child childFilter='field:value']. There was no escaping here prior to 8.0 either.
Collapse and Expand
BlockCollapse: If documents have been (or could be) indexed in a way where documents with the same collapse key have been indexed contiguously in the index, a new "block collapse" provides a significant speed improvement over traditional collapse.
See Block Collapsing for details.
Expand Null Groups: A new parameter expand.nullGroup allows an expanded group to be returned containing document with no value in the expanded field. See Expand Component for details.
In-Place Updates
A new request parameter update.partial.requireInPlace=true allows telling Solr to "fail fast" if all of the necessary conditions are not satisfied to allow an in-place update to succeed. See also In-Place Updates.
Metrics History
The Metrics History feature, which allowed long-term storage and aggregation of Solr’s metrics, has been deprecated and will be removed in 9.0.
Embedded Solr Server
When using EmbeddedSolrServer, it will no longer close CoreContainer instances that were passed to it.
Solr 8.8
When upgrading to 8.8.x users should be aware of the following major changes from 8.7.
Nested Documents
When doing atomic/partial updates to a child document:
Supply the _root_ field (the ID of the root document) so that Solr understands you are manipulating a child document and not a root document. In its absence, Solr looks at the _route_ parameter but this may change in the future because it’s not an ideal substitute. If neither are present, Solr assumes you are updating a root document. If this assumption is false, Solr will do a cheap check that usually detects the problem and will throw an exception to alert you of the need to specify the Root ID. This backwards incompatible change was done to increase performance and robustness.
This feature no longer requires stored=true or docValues=true on the _root_ field. You might have it for other purposes though (e.g., for uniqueBlock(…)).
This feature no longer requires the _nest_path_ field, although you probably ought to continue to define it as it’s useful for other things.
Removed Contribs
The search results clustering contrib (Carrot2) has been removed from 8.x Solr due to lack of Java 1.8 compatibility in the dependency that provides online clustering of search results. The contrib will be re-introduced in Solr 9.0.
Learning to Rank
Interleaving support has been added to Learning to Rank (LTR). Currently only the Team Draft Interleaving algorithm is supported. For examples using this feature, see the section Running a Rerank Query Interleaving Two Models.
Metrics
Two metrics have been added for SolrCloud’s Overseer:
solr_metrics_overseer_stateUpdateQueueSize
solr_metrics_overseer_collectionWorkQueueSize
Prometheus Exporter
The ./bin scripts included with the Prometheus Exporter contrib now allow use of custom java options with environment variables. See the section Environment Variable Options for more details.
The default Grafana dashboards now include panels for query performance monitoring. The default Prometheus Exporter configuration includes metrics like queries-per-second (QPS) and 95th percentiles (P95) to populate the new panels.
The default Prometheus Exporter configuration also includes the two new metrics mentioned in the Metrics above.
Solr Home
The internal logic for identifying 'Solr Home' (solr.solr.home) has been refactored to make testing less error prone. Plugin developers using SolrPaths.locateSolrHome() or 'new `SolrResourceLoader’ should check deprecation warnings as existing some existing functionality will be removed in 9.0. SOLR-14934 has more technical details about this change for those concerned.
base_url removed from stored state
As of Solr 8.8.0, the base_url property was removed from the stored state for replicas (SOLR-12182). If you’re able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set -Dsolr.storeBaseUrl=false (introduced in Solr 8.8.1) to better align the stored state in ZooKeeper with future versions of Solr. However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then leave the default -Dsolr.storeBaseUrl=true so that Solr will continue to store the base_url in ZooKeeper.
You may also see some NPE in collection state updates during a rolling upgrade to 8.8.0 from a previous version of Solr. After upgrading all nodes in your cluster to 8.8.0, collections should fully recover. Trigger another rolling restart if there are any replicas that do not recover after the upgrade to re-elect leaders.
Solr 8.7
See the 8.7 Release Notes for an overview of the main new features of Solr 8.7.
When upgrading to 8.7.x users should be aware of the following major changes from 8.6.
Autoscaling
If upgrading from 8.6.0, please see the 8.6.1 Upgrade notes below for information on performance degradations introduced in 8.6.0 that require some intervention to resolve. If you are already on 8.6.1 or higher, you can ignore these instructions.
Deprecations
The autoscaling framework is now formally deprecated and will be removed in Solr 9.0. The Solr community is working on pluggable API to replace this functionality, with the goal for it to be ready by the time 9.0 is released. Deprecations include: autoscaling policy, triggers, withCollection support, simulation framework, autoscaling suggestions tab in the UI, autoAddReplicas and UTILIZENODE command.
Similarly, rule-based replica placement strategy has been deprecated and will be replaced in Solr 9.0 by APIs for replica placement and cluster events, with plugin-based implementations.
Support for detecting spinning disks has been removed in LUCENE-9576. Corresponding spins metrics in Solr still exist but now they always return false and will be removed in Solr 9.0.
User-Managed Cluster Terminology Updated
Solr has replaced the terms "master" and "slave" in the codebase and all documentation with "leader" and "follower".
This functionality has only changed in terms of parameter names changed, and we do not expect any back-compatibility issues on upgrade to 8.7 or even 9.0 later.
However, users should update their solrconfig.xml files after completing the upgrade on all nodes of a cluster. Comparing your configuration to the updated configuration examples in Index Replication will show examples of what needs to change, but here are the main changes:
On the replication leader, in the definition of the /replication request handler:
Replace "master" with "leader".
Replace "slave" with "follower" if the former term is used in the name of any follower solrconfig.xml file definitions. This file can be named anything, so you can change it to whatever you’d like to call it if you’d like.
Replace "slave" with "follower" if the former term is used in a replication repeater configuration.
On the replication follower, in the definition of the /replication request handler:
Replace "masterUrl" with "leaderUrl".
Replace "slave" with "follower" if the former term is used in a repeater configuration.
JSON Facets
Performance enhancements for the relatedness() statistics function are included with 8.7. These yield the highest benefits with high-cardinality fields and can be disabled if working with lower cardinality fields with a new sweep_collection parameter. See the section relatedness() Options for details.
solr.in.sh / solr.in.cmd
Solr has relied on the SOLR_STOP_WAIT parameter defined in solr.in.sh or solr.in.cmd to determine how long to wait on startup. A new parameter SOLR_START_WAIT allows defining how long Solr should wait for start up to complete.
If the time set by this parameter is exceeded, Solr will exit the startup process and return the last few lines of the solr.log file to the terminal.
By default, this parameter is set to the same value as SOLR_STOP_WAIT.
The default ZooKeeper client timeout (ZK_CLIENT_TIMEOUT) is now 30 seconds (30000 milliseconds) instead of 15.
Configsets
It’s now possible to overwrite an existing configset when uploading changes by supplying the overwrite=true parameter to the Configset API.
A related parameter is cleanup=true, which allows deleting any files from the old configset that are left behind after the overwrite.
The default for both of these parameters is false.
When deleting a collection that has an automatically created configset (i.e., the configset was copied from the _default collection when the collection was created), the configset will also be deleted if it is not in use by any other collection.
Logging
A request ID (rid) is now logged for all distributed search requests (in SolrCloud) which can be used to correlate query events across the system. A parameter disableRequestId=true can be added to disable this if desired.
Solr 8.6.1
See the 8.6.1 Release Notes for an overview of the fixes included in Solr 8.6.1.
When upgrading to 8.6.1 users should be aware of the following major changes from 8.6.0.
Autoscaling
As mentioned in the 8.6 upgrade notes, a default autoscaling policy was provided starting in 8.6.0. This default autoscaling policy resulted in increasingly slow collection creation calls in large clusters (50+ collections).
In 8.6.1 the default autoscaling policy has been removed, and clusters will not use autoscaling unless a policy has explicitly been created. If your cluster is running 8.6.0 and not using an explicit autoscaling policy, upgrade to 8.6.1 and remove the default cluster policy and preferences via the following command.
Replace localhost:8983 with your Solr endpoint.
curl -X POST -H 'Content-type:application/json' -d '{set-cluster-policy : [], set-cluster-preferences : []}' http://localhost:8983/api/cluster/autoscaling
This information is only relevant for users upgrading from 8.6.0. If upgrading from an earlier version to 8.6.1+, this warning can be ignored.
Solr 8.6
See the 8.6 Release Notes for an overview of the main new features of Solr 8.6.
When upgrading to 8.6.x users should be aware of the following major changes from 8.5.
Support for Block-Max WAND
Lucene added support for Block-Max WAND in 8.0, and 8.6 makes this available for Solr also.
This can provide significant performance enhancements by not calculating the score for results which are not likely to appear in the top set of results.
It is enabled when using a new query parameter minExactCount. This parameter tells Solr to accurately count the number of hits accurately until at least this value. Once this value is reached, Solr can skip over documents that don’t have a score high enough to be in the top set of documents, which has the potential for greatly speeding up searches.
It’s important to note that when using this parameter, the hit count of searches may not be accurate. It is guaranteed that the hit count is accurate up to the value of minExactCount, but any returned hit count higher than that may be an approximation.
A new boolean attribute numFoundExact is included in all responses to indicate if the hit count in the response is expected to be exact or not.
More information about this new feature is available in the section minExactCount Parameter.
Autoscaling
NOTE: The default autoscaling policy has been removed as of 8.6.1
Solr now includes a default autoscaling policy. This policy can be overridden with your custom rules or by specifying an empty policy to replace the default.
The ComputePlan action now supports a collection selector to identify collections based on collection properties to determine which collections should be operated on.
Security
Prior to Solr 8.6 Solr APIs which take a file system location, such as core creation, backup, restore, and others, did not validate the path and Solr would allow any absolute or relative path. Starting in 8.6 only paths that are relative to SOLR_HOME, SOLR_DATA_HOME and coreRootDir are allowed by default.
If you need to create a core or store a backup outside the default paths, you will need to tell Solr which paths to allow. A new element in solr.xml called allowPaths takes a comma-separated list of allowed paths.
When using the solr.xml file that ships with 8.6, you can configure the list of paths to allow through the system property solr.allowPaths. Please see bin/solr.in.sh or bin\solr.in.cmd for example usage. Using the value * will allow any path as in earlier versions.
For more on this, see the section Solr.xml Parameters.
Windows SMB shares on the UNC format, such as \\myhost\myshare\mypath are now always disallowed. Please use drive letter mounts instead, i.e., S:\mypath.
A new authorization plugin ExternalRoleRuleBasedAuthorizationPlugin is now available. This plugin allows an authentication plugin (such as JWT) to supply a user’s roles instead of maintaining a user-to-role mapping inside Solr.
When authentication is enabled, the Admin UI Dashboard (main screen) now includes a panel that shows the authentication and authorization plugins in use, the logged in username, and the roles assigned to this user. A new link will also appear in the left-hand navigation to allow a user to log out.
Streaming Expressions
The /export handler now supports streaming expressions to allow limiting the output of the export to only matching documents.
For more information about how to use this, see the section Specifying the Local Streaming Expression.
The stats, facet, and timeseries expressions now support percentiles and standard deviation aggregations.
Highlighting
For the Unified Highlighter: The setting hl.fragsizeIsMinimum now defaults to false because true was found to be a significant performance regression when highlighting lots of text. This will yield longer highlights on average compared to Solr 8.5 but relatively unchanged compared to previous releases. Furthermore, if your application highlights lots of text, you may want to experiment with lowering hl.fragAlignRatio to trade ideal fragment alignment for better performance.
Deprecations
A primary focus of the community is improving Solr’s stability and supportability. With the addition of the package manager system in 8.4, we now have the ability to move some features into plugins maintained by third-parties with the expertise to properly develop and support them. Our goal is to make running Solr easier and less prone to outages and other headaches. In this spirit, the following features have been deprecated and are slated to be removed in Solr 9.0.
Cross Data Center Replication (CDCR), in its current form, is deprecated and is scheduled to be removed in 9.0. This feature is unlikely to be replaced by an identical plugin. However, the community is working on figuring out a replacement feature for disaster recovery and failover.
The Data Import Handler (DIH) is deprecated and is scheduled to be removed in 9.0. Work to replace DIH with a community-supported plugin is underway and may be available soon.
Support to store indexes and backups in HDFS is deprecated and is scheduled to be removed in 9.0. A community-supported version of this may be available as a plugin in the future. For more details, please see SOLR-14021.
Users interested in maintaining a feature as a plugin are encouraged to join the developer mailing list to find out more about how to help.
See NEWS in sources for details. A summary of upstream NEWS:
* Enhancements and bugfixes
PostGIS 3.2.0 (Olivier Courtin Edition)
2021/12/17
This version requires PostgreSQL 9.6 or higher, GEOS 3.6 or higher
, and Proj 4.9+.
Additional features are enabled if you are running GEOS 3.9+
(and ST_MakeValid enhancements with 3.10+),
Proj 6.1+, and PostgreSQL 14+.
Due to some query performance degradation
with the new fast index build that requires PG14,
we have decided to disable the feature by default
until we get more user testing
as to the true impact of real-world queries.
If you are running PG14+, you can reenable it by doing
ALTER OPERATOR FAMILY gist_geometry_ops_2d USING gist
ADD FUNCTION 11 (geometry)
geometry_gist_sortsupport_2d (internal);
and then reindex your gist indexes.
To revert back to the old index behavior:
ALTER OPERATOR FAMILY gist_geometry_ops_2d using gist
DROP FUNCTION 11 (geometry);
and then reindex your gist indexes.
* Breaking changes *
- #5008, Empty geometries are not reported as being within Infinite
distance by ST_DWithin (Sandro Santilli)
- #4824, Removed `--without-wagyu` build option. Using Wagyu is now mandatory to build with MVT support.
- #4933, topology.GetFaceByPoint will not work with topologies having invalid edge linking.
- #4981, ST_StartPoint support any geometry. No longer returns null for non-linestrings.
- #4149, ST_AsMVTGeom now preserves more of original geometry's details at scale close to target extent.
If you need previous simplifying behaviour, you can ST_Simplify the geometry in advance.
(Darafei Praliaskouski)
- Proj 4.9 or higher is required
- #5000, Turn off Window support in ST_AsMVT aggregate
as no real use-case for it and it crashes with random input
(Paul Ramsey)
* New features*
- #4923, topology.ValidateTopologyRelation (Sandro Santilli)
- #4933, topology.GetFaceContainingPoint (Sandro Santilli)
- #2175, ST_Scroll (Sandro Santilli)
- #4841, FindTopology to quickly get a topology record (Sandro Santilli)
- #4869, FindLayer to quickly get a layer record (Sandro Santilli)
- #4851, TopoGeom_addTopoGeom function (Sandro Santilli)
- ST_MakeValid(geometry, options) allows alternative validity building
algorithms with GEOS 3.10 (Paul Ramsey)
- ST_InterpolateRaster() fills in raster cells between sample points
using one of a number of algorithms (inverse weighted distance, average, etc)
using algorithms from GDAL
(Paul Ramsey)
- ST_Contour() generates contour lines from raster values
using algorithms from GDAL (Paul Ramsey)
- ST_SetZ()/ST_SetM() fills in z/m coordinates of a geometry using data read
from a raster (Paul Ramsey)
- New postgis.gdal_vsi_options GUC allows out-db rasters on VSI network
services to be accessed with authentication keys, etc. (Paul Ramsey)
- ST_DumpSegments returns a set of segments of input geometry (Aliaksandr Kalenik)
- #4859, ST_Point, ST_PointZ, ST_PointM, ST_PointZM, constructors
with SRID parameter (Paul Ramsey)
- #4808, ST_ClusterKMeans now supports max_radius argument. Use it when you're not sure what is
the number of clusters but you know what the size of clusters should be. (Darafei Praliaskouski)
dbplyr is the database backend for dplyr. It allows you to use
remote database tables as if they are in-memory data frames by
automatically converting dplyr code into SQL.
Breaks build with python 2.7 because the paths are different there.
Still packages with python 3.6, 3.9, 3.10 on NetBSD.
Bump PKGREVISION for PLIST change.
Upstream changes:
0.143 2021-03-28
- fix RT #134147: Failed test 'SQL::Abstract - no limit' / Thanks to Slaven Rezić and Thibault Duponchelle
0.142 2020-11-22
- module adopted by Alexander Becker
- set up GitHub repository. See this repository for detailed changes.
- fix RT #34847: Avoid double quoting of table name. Fix provided by @dekimsey / Daniel Kimsey
- fix RT #50795: SQL::Abstract properties were ignored for select() if no limit was given. Thanks to Gunnar Hansson
- fix RT #73444: Offset argument wasn't handled properly if the value was 0. Thanks to ALEXBYK
- fix RT #62671: Properly handle 0 values in condition.
- remove Build.PL, switch to EUMM
Avoids build failure if monogb 4.x is installed.
Someone can always add a mongodb option to options.mk if they want, but
this is just to ensure reliable builds
- Updated all the live engine tests, aside from Oracle, to test with
unique registry names, so as to avoid conflicts when multiple
instances are being tested at once. Thanks to Slaven Rezić for the
report (#597).
- Removed `local` directory from the distribution, accidentally included
in v1.2.0. Thanks to gregor herrmann for the report (#600).
5.51.0 (2021-12-01)
* Make eager loading via tactical_eager_loading no longer modify objects who
already have a cached value for the association (jeremyevans)
* Make association cloning handle cases where clone association sets
different :class option than cloned association (jeremyevans)
* Make column schema entries on MySQL include an :extra entry for the Extra
column in DESCRIBE output (bschmeck) (#1791)
* Update mock PostgreSQL adapter to default to PostgreSQL 14 instead of
PostgreSQL 9.5 (jeremyevans)
* Support Dataset#with_recursive :search and :cycle options on PostgreSQL
14+ for result ordering and cycle detection (jeremyevans)
* Avoid method redefined verbose mode warnings in lazy_attributes plugin
(jeremyevans)
Version 1.3.0 - 2021-09-14
* Add GROUPS frame to Window
* Add exclude to Window
* Add method for each type of join
* Remove support for Python older than 3.5
* Support Select queries in conditionals and functions
Version 1.2.2 - 2021-05-16
* Escape table and column names containing double quote
* Use alias on the main table used in UPDATE and INSERT
* Add support for Python 3.9
Version 1.2.1 - 2020-10-08
* Fix AttributeError in Select.params with windows
Version 1.2.0 - 2020-10-05
* Add explicit Windows to Select
* Fix missing Windows definitions in nested expressions
Version 1.1.0 - 2020-01-25
* Add ORDER BY clause to aggregate functions
* Add support for Python 3.8
* Add distinct on Select
Version 1.0.0 - 2018-09-30
* Add Flavor filter_ to fallback to case expression
* Allow to use expression in AtTimeZone
* Fix Select query in returning
* Add comparison predicates
* Add COLLATE
kVS: ----------------------------------------------------------------------
SQLite Release 3.37.0 On 2021-11-27
STRICT tables provide a prescriptive style of data type management, for developers who prefer that kind of thing.
When adding columns that contain a CHECK constraint or a generated column containing a NOT NULL constraint, the ALTER TABLE ADD COLUMN now checks new constraints against preexisting rows in the database and will only proceed if no constraints are violated.
Added the PRAGMA table_list statement.
CLI enhancements:
Add the .connection command, allowing the CLI to keep multiple database connections open at the same time.
Add the --safe command-line option that disables dot-commands and SQL statements that might cause side-effects that extend beyond the single database file named on the command-line.
Performance improvements when reading SQL statements that span many lines.
Added the sqlite3_autovacuum_pages() interface.
The sqlite3_deserialize() does not and has never worked for the TEMP database. That limitation is now noted in the documentation.
The query planner now omits ORDER BY clauses on subqueries and views if removing those clauses does not change the semantics of the query.
The generate_series table-valued function extension is modified so that the first parameter ("START") is now required. This is done as a way to demonstrate how to write table-valued functions with required parameters. The legacy behavior is available using the -DZERO_ARGUMENT_GENERATE_SERIES compile-time option.
Added new sqlite3_changes64() and sqlite3_total_changes64() interfaces.
Added the SQLITE_OPEN_EXRESCODE flag option to sqlite3_open_v2().
Use less memory to hold the database schema.
Currently supports dBase III, Clipper, FoxPro, and Visual FoxPro
tables. Text is returned as unicode, and codepage settings in tables
are honored. Memos and Null fields are supported. Documentation
needs work, but author is very responsive to e-mails.
Not supported: index files (but can create tempory non-file indexes),
auto-incrementing fields, and Varchar fields.
- Fixed partitioned search for deployed changes on Oracle and Exasol to
correctly support plans with more than 250 changes. Thanks to @Nicqu
for the report (#521).
- DBI connections to the registry will now be set to trace level one
when a verbosity level of three or higher is passed to Sqitch
(i.e., `sqitch -vvv`). Thanks to @wkoszek for the suggestion (#155).
- Renamed the "master" branch to "main" and updated all relevant
references.
- Fixed the parsing of the Snowflake account name from the target URL
to that it no longer strips out the region and cloud platform parts.
Also deprecated the use of the region, host, and port config and
environment variables. This is to comply with the new account name
format. Thanks to @ldsingh00 for the report (#544).
- The checkout command will now show a usage statement when no branch
name is passed to it, rather than a series of warnings and a Git error.
Thanks to François Beausoleil for the report (#561).
- The checkout command now works when operating on a Sqitch project in
a subdirectory of a Git project. Thanks to François Beausoleil for the
report and suggested fix (#560).
- Fixed a failing bundle test when a top directory was configured in the
user or system configuration. Thanks to @CodingMinds for the spot
(#587).
- Added support to the Exasol engine for the `AUTHMETHOD` query parameter
(to allow Open ID authentication) and the `SSLCERTIFICATE=SSL_VERIFY_NONE`
query parameter to disable SSL verification. Thanks to Torsten Glunde
for the report (#588).
- Fixed "Use of uninitialized value $engine in concatenation" error when
Sqitch is unable to determine the engine when writing a plan file. Only
happens in the rare case that the core.engine value is not set.
- Improved the error message when attempting to update a plan file and no
project pragma is present. Thanks to Laurentiu Diaconu for the report
(#577).
- Fixed the error message when attempting to deploy a change that has
already been deployed to display the name of the change, rather than a
memory address. Thanks to Neil Mayhew for the report (#579).
- Added destination locking, currently implemented for PostgresQL and
MySQL. On starting a deploy or revert, Sqitch attempts to "lock the
destination" using advisory locks, to ensure that only one instance of
Sqitch makes changes to the database at any one time. This complements
the existing locking, which applies as each change is deployed or
reverted, as that pattern led to failures when multiple instances of
Sqitch were working at once. Thanks to Neil Mayhew for the report
(#579).
- Added the --lock-timeout option to the deploy, revert, rebase, and
checkout commands, to allow for shorter or longer times to wait for an
exclusive target database advisory lock for engines that support it.
Defaults to 60 seconds.
- Fixed the behavior of the plan command's `--max-count` option when used
with `--reverse` to show the proper items. Thanks to Adrian Klaver for
the report (#517).
- Fixed an issue that could cause bugs with the `check` command on
Firebird, Exasol, Oracle, and Vertica. Broken since the `check` command
was added in v1.1.0.
v0.25.0
Changes
Improve SSL option compatibility in URIs
Add Pool methods to determine its min, max, current and idle size
Make it possible to specify a statement name in Connection.prepare()
Implement support for multirange types
Fixes
Make sure timeout callbacks always get cleaned up
Update __all__ statements to a simpler form that is better supported by typecheckers
Fix test_timetz_encoding on Python 3.10
Fix a bunch of ResourceWarnings in the test suite
Fix SSLContext deprecation warnings
Fix the description of the database argument to connect()
Fix parsing of IPv6 addresses in the connection URI
Improve diagnostics of invalid executemany() input
PostgreSQL 14.1, 13.5, 12.9, 11.14, 10.19, and 9.6.24
Security Issues
CVE-2021-23214: Server processes unencrypted bytes from man-in-the-middle
Versions Affected: 9.6 - 14. The security team typically does not test unsupported versions, but this problem is quite old.
When the server is configured to use trust authentication with a clientcert requirement or to use cert authentication, a man-in-the-middle attacker can inject arbitrary SQL queries when a connection is first established, despite the use of SSL certificate verification and encryption.
The PostgreSQL project thanks Jacob Champion for reporting this problem.
CVE-2021-23222: libpq processes unencrypted bytes from man-in-the-middle
Versions Affected: 9.6 - 14. The security team typically does not test unsupported versions, but this problem is quite old.
A man-in-the-middle attacker can inject false responses to the client's first few queries, despite the use of SSL certificate verification and encryption.
If more preconditions hold, the attacker can exfiltrate the client's password or other confidential data that might be transmitted early in a session. The attacker must have a way to trick the client's intended server into making the confidential data accessible to the attacker. A known implementation having that property is a PostgreSQL configuration vulnerable to CVE-2021-23214.
As with any exploitation of CVE-2021-23214, the server must be using trust authentication with a clientcert requirement or using cert authentication. To disclose a password, the client must be in possession of a password, which is atypical when using an authentication configuration vulnerable to CVE-2021-23214. The attacker must have some other way to access the server to retrieve the exfiltrated data (a valid, unprivileged login account would be sufficient).
The PostgreSQL project thanks Jacob Champion for reporting this problem.
Bug Fixes and Improvements
This update fixes over 40 bugs that were reported in the last several months. The issues listed below affect PostgreSQL 14. Some of these issues may also affect other supported versions of PostgreSQL.
Some of these fixes include:
Fix physical replication for cases where the primary crashes after shipping a WAL segment that ends with a partial WAL record. When applying this update, update your standby servers before the primary so that they will be ready to handle the fix if the primary happens to crash.
Fix parallel VACUUM so that it will process indexes below the min_parallel_index_scan_size threshold if the table has at least two indexes that are above that size. This problem does not affect autovacuum. If you are affected by this issue, you should reindex any manually-vacuumed tables.
Fix causes of CREATE INDEX CONCURRENTLY and REINDEX CONCURRENTLY writing corrupt indexes. You should reindex any concurrently-built indexes.
Fix for attaching/detaching a partition that could allow certain INSERT/UPDATE queries to misbehave in active sessions.
Fix for creating a new range type with CREATE TYPE that could cause problems for later event triggers or subsequent executions of the CREATE TYPE command.
Fix updates of element fields in arrays of a domain that is a part of a composite.
Disallow the combination of FETCH FIRST WITH TIES and FOR UPDATE SKIP LOCKED.
Fix corner-case loss of precision in the numeric power() function.
Fix restoration of a Portal's snapshot inside a subtransaction, which could lead to a crash. For example, this could occur in PL/pgSQL when a COMMIT is immediately followed by a BEGIN ... EXCEPTION block that performs a query.
Clean up correctly if a transaction fails after exporting its snapshot. This could occur if a replication slot was created then rolled back, and then another replication slot was created in the same session.
Fix for "overflowed-subtransaction" wraparound tracking on standby servers that could lead to performance degradation.
Ensure that prepared transactions are properly accounted for during promotion of a standby server.
Ensure that the correct lock level is used when renaming a table.
Avoid crash when dropping a role that owns objects being dropped concurrently.
Disallow setting huge_pages to on when shared_memory_type is sysv
Fix query type checking in the PL/pgSQL RETURN QUERY.
Several fixes for pg_dump, including the ability to dump non-global default privileges correctly.
Use the CLDR project's data to map Windows time zone names to IANA time zones.
This update also contains tzdata release 2021e for DST law changes in Fiji, Jordan, Palestine, and Samoa, plus historical corrections for Barbados, Cook Islands, Guyana, Niue, Portugal, and Tonga.
Also, the Pacific/Enderbury zone has been renamed to Pacific/Kanton. Also, the following zones have been merged into nearby, more-populous zones whose clocks have agreed with them since 1970: Africa/Accra, America/Atikokan, America/Blanc-Sablon, America/Creston, America/Curacao, America/Nassau, America/Port_of_Spain, Antarctica/DumontDUrville, and Antarctica/Syowa. In all these cases, the previous zone name remains as an alias.
1.7.5
Released: November 11, 2021
bug
[bug] [tests]
Adjustments to the test suite to accommodate for error message changes occurring as of SQLAlchemy 1.4.27.
1.7.4
Released: October 6, 2021
bug
[bug] [regression]
Fixed a regression that prevented the use of post write hooks on python version lower than 3.9
[bug] [environment]
Fixed issue where the MigrationContext.autocommit_block() feature would fail to function when using a SQLAlchemy engine using 2.0 future mode.
1.7.3
Released: September 17, 2021
bug
[bug] [mypy]
Fixed type annotations for the “constraint_name” argument of operations create_primary_key(), create_foreign_key(). Pull request courtesy TilmanK.
1.7.2
Released: September 17, 2021
bug
[bug] [typing]
Added missing attributes from context stubs.
[bug] [mypy]
Fixed an import in one of the .pyi files that was triggering an assertion error in some versions of mypy.
[bug] [ops] [regression]
Fixed issue where registration of custom ops was prone to failure due to the registration process running exec() on generated code that as of the 1.7 series includes pep-484 annotations, which in the case of end user code would result in name resolution errors when the exec occurs. The logic in question has been altered so that the annotations are rendered as forward references so that the exec() can proceed.
1.7.1
Released: August 30, 2021
bug
[bug] [installation]
Corrected “universal wheel” directive in setup.cfg so that building a wheel does not target Python 2. The PyPi files index for 1.7.0 was corrected manually. Pull request courtesy layday.
[bug] [pep484]
Fixed issue in generated .pyi files where default values for Optional arguments were missing, thereby causing mypy to consider them as required.
[bug] [batch] [regression]
Fixed regression in batch mode due to 883 where the “auto” mode of batch would fail to accommodate any additional migration directives beyond encountering an add_column() directive, due to a mis-application of the conditional logic that was added as part of this change, leading to “recreate” mode not being used in cases where it is required for SQLite such as for unique constraints.
1.7.0
Released: August 30, 2021
changed
[changed] [installation]
Alembic 1.7 now supports Python 3.6 and above; support for prior versions including Python 2.7 has been dropped.
[changed] [installation]
Make the python-dateutil library an optional dependency. This library is only required if the timezone option is used in the Alembic configuration. An extra require named tz is available with pip install alembic[tz] to install it.
[changed] [installation]
The dependency on pkg_resources which is part of setuptools has been removed, so there is no longer any runtime dependency on setuptools. The functionality has been replaced with importlib.metadata and importlib.resources which are both part of Python std.lib, or via pypy dependency importlib-metadata for Python version < 3.8 and importlib-resources for Python version < 3.9 (while importlib.resources was added to Python in 3.7, it did not include the “files” API until 3.9).
feature
[feature] [environment]
Enhance version_locations parsing to handle paths containing spaces. The new configuration option version_path_separator specifies the character to use when splitting the version_locations string. The default for new configurations is version_path_separator = os, which will use os.pathsep (e.g., ; on Windows).
[feature] [tests]
Created a “test suite” similar to the one for SQLAlchemy, allowing developers of third-party dialects to test their code against a set of Alembic tests that have been specially selected to exercise back-end database operations. At the time of release, third-party dialects that have adopted the Alembic test suite to verify compatibility include CockroachDB and SAP ASE (Sybase).
[feature] [general]
pep-484 type annotations have been added throughout the library. Additionally, stub .pyi files have been added for the “dynamically” generated Alembic modules alembic.op and alembic.config, which include complete function signatures and docstrings, so that the functions in these namespaces will have both IDE support (vscode, pycharm, etc) as well as support for typing tools like Mypy. The files themselves are statically generated from their source functions within the source tree.
usecase
[usecase] [batch]
Named CHECK constraints are now supported by batch mode, and will automatically be part of the recreated table assuming they are named. They also can be explicitly dropped using op.drop_constraint(). For “unnamed” CHECK constraints, these are still skipped as they cannot be distinguished from the CHECK constraints that are generated by the Boolean and Enum datatypes.
Note that this change may require adjustments to migrations that drop or rename columns which feature an associated named check constraint, such that an additional op.drop_constraint() directive should be added for that named constraint as there will no longer be an associated column for it; for the Boolean and Enum datatypes, an existing_type keyword may be passed to BatchOperations.drop_constraint as well.
See also Changing the Type of Boolean, Enum and other implicit CHECK datatypes
Including CHECK constraints
bug
[bug] [operations]
Fixed regression due to 803 where the .info and .comment attributes of Table would be lost inside of the DropTableOp class, which when “reversed” into a CreateTableOp would then have lost these elements. Pull request courtesy Nicolas CANIART.
[bug] [batch] [sqlite]
Batch “auto” mode will now select for “recreate” if the add_column() operation is used on SQLite, and the column itself meets the criteria for SQLite where ADD COLUMN is not allowed, in this case a functional or parenthesized SQL expression or a Computed (i.e. generated) column.
[bug] [commands]
Re-implemented the python-editor dependency as a small internal function to avoid the need for external dependencies.
[bug] [postgresql]
Fixed issue where usage of the PostgreSQL postgresql_include option within a Operations.create_index() would raise a KeyError, as the additional column(s) need to be added to the table object used by the construct internally. The issue is equivalent to the SQL Server issue fixed in 513. Pull request courtesy Steven Bronson.
1.6.5
Released: May 27, 2021
bug
[bug] [autogenerate]
Fixed issue where dialect-specific keyword arguments within the DropIndex operation directive would not render in the autogenerated Python code. As support was improved for adding dialect specific arguments to directives as part of 803, in particular arguments such as “postgresql_concurrently” which apply to the actual create/drop of the index, support was needed for these to render even in a drop index operation. Pull request courtesy Jet Zhou.
1.6.4
Released: May 24, 2021
bug
[bug] [op directives] [regression]
Fixed regression caused by just fixed 844 that scaled back the filter for unique=True/index=True too far such that these directives no longer worked for the op.create_table() op, this has been fixed.
1.6.3
Released: May 21, 2021
bug
[bug] [autogenerate] [regression]
Fixed 1.6-series regression where UniqueConstraint and to a lesser extent Index objects would be doubled up in the generated model when the unique=True / index=True flags were used.
[bug] [autogenerate]
Fixed a bug where paths defined in post-write hook options would be wrongly escaped in non posix environment (Windows).
[bug] [regression] [versioning]
Fixed regression where a revision file that contained its own down revision as a dependency would cause an endless loop in the traversal logic.
1.6.2
Released: May 6, 2021
bug
[bug] [regression] [versioning]
Fixed additional regression nearly the same as that of 838 just released in 1.6.1 but within a slightly different codepath, where “alembic downgrade head” (or equivalent) would fail instead of iterating no revisions.
1.6.1
Released: May 6, 2021
bug
[bug] [regression] [versioning]
Fixed regression in new revisioning traversal where “alembic downgrade base” would fail if the database itself were clean and unversioned; additionally repairs the case where downgrade would fail if attempting to downgrade to the current head that is already present.
1.6.0
Released: May 3, 2021
feature
[feature] [autogenerate]
Fix the documentation regarding the default command-line argument position of the revision script filename within the post-write hook arguments. Implement a REVISION_SCRIPT_FILENAME token, enabling the position to be changed. Switch from str.split() to shlex.split() for more robust command-line argument parsing.
[feature]
Implement a .cwd (current working directory) suboption for post-write hooks (of type console_scripts). This is useful for tools like pre-commit, which rely on the working directory to locate the necessary config files. Add pre-commit as an example to the documentation. Minor change: rename some variables from ticket 819 to improve readability.
bug
[bug] [autogenerate]
Refactored the implementation of MigrateOperation constructs such as CreateIndexOp, CreateTableOp, etc. so that they no longer rely upon maintaining a persistent version of each schema object internally; instead, the state variables of each operation object will be used to produce the corresponding construct when the operation is invoked. The rationale is so that environments which make use of operation-manipulation schemes such as those those discussed in Fine-Grained Autogenerate Generation with Rewriters are better supported, allowing end-user code to manipulate the public attributes of these objects which will then be expressed in the final output, an example is some_create_index_op.kw["postgresql_concurrently"] = True.
Previously, these objects when generated from autogenerate would typically hold onto the original, reflected element internally without honoring the other state variables of each construct, preventing the public API from working.
[bug] [environment]
Fixed regression caused by the SQLAlchemy 1.4/2.0 compatibility switch where calling .rollback() or .commit() explicitly within the context.begin_transaction() context manager would cause it to fail when the block ended, as it did not expect that the transaction was manually closed.
[bug] [autogenerate]
Improved the rendering of op.add_column() operations when adding multiple columns to an existing table, so that the order of these statements matches the order in which the columns were declared in the application’s table metadata. Previously the added columns were being sorted alphabetically.
[bug] [versioning]
The algorithm used for calculating downgrades/upgrades/iterating revisions has been rewritten, to resolve ongoing issues of branches not being handled consistently particularly within downgrade operations, as well as for overall clarity and maintainability. This change includes that a deprecation warning is emitted if an ambiguous command such as “downgrade -1” when multiple heads are present is given.
In particular, the change implements a long-requested use case of allowing downgrades of a single branch to a branchpoint.
Huge thanks to Simon Bowly for their impressive efforts in successfully tackling this very difficult problem.
[bug] [batch] ¶
Added missing batch_op.create_table_comment(), batch_op.drop_table_comment() directives to batch ops.
DBD::MariaDB is the Perl5 Database Interface driver for MariaDB
and MySQL databases. In other words: DBD::MariaDB is an interface
between the Perl programming language and the MariaDB/MySQL
programming API that comes with the MariaDB/MySQL relational database
management system. Most functions provided by this programming API
are supported. Some rarely used functions are missing, mainly
because no-one ever requested them. :-)
=== 5.50.0 (2021-11-01)
* Make Migrator :allow_missing_migration_files also allow down migrations
where the current database version is greater than the last migration file
version (francisconeves97) (#1789)
* Fix Model#freeze in composition, serialization, and
serialization_modification_detection plugins to return self (jeremyevans)
(#1788)
* Fix typecasting of lazy columns when using lazy_attributes plugin in model
where dataset selects from subquery (jeremyevans)
* Add :before_preconnect Database option, for configuring extensions loaded
via :preconnect_extensions (MarcPer, jeremyevans) (#1786)
* Change Dataset#columns! to use a LIMIT 0 query instead of a LIMIT 1 query
(jeremyevans)
* Add sql_log_normalizer extension for normalizing logged SQL, helpful for
analytics and sensitive data (jeremyevans)
* Add support for range_merge, multirange, and unnest, and PGMultiRange#op
to pg_range_ops extension (jeremyevans)
* Add pg_multirange extension with support for PostgreSQL 14+ multirange
types (jeremyevans)
=== 5.49.0 (2021-10-01)
* Switch block_given? usage to defined?(yield) (jeremyevans)
* Support table aliases for JOIN USING columns on PostgreSQL 14+
(jeremyevans)
* Support calling PostgreSQL procedures without arguments (jeremyevans)
* Support hstore subscripts in pg_hstore_ops on PostgreSQL 14+, for updating
only part of an hstore value (jeremyevans)
* Support JSONB subscripts in pg_json_ops on PostgreSQL 14+, for updating
only part of a JSONB value (jeremyevans)
* Support SQL::Expression#sequel_ast_transform for custom AST transforms on
arbitrary expressions (jeremyevans)
* Add Database#create_trigger :replace option on PostgreSQL 14+ for CREATE
OR REPLACE TRIGGER (jeremyevans)
* Make auto_validations plugin automatically setup no_null_byte validations
(jeremyevans)
* Add Model#validates_no_null_byte to validation_helpers plugin
(jeremyevans)
A PostgreSQL SQL syntax beautifier that can work as a console program or as a
CGI.
This SQL formatter/beautifier supports keywords from SQL-92, SQL-99, SQL-2003,
SQL-2008, SQL-2011 and PostgreSQL specifics keywords. Works with any other
databases too.
Changes from the submitted version in the PR:
- p5 prefix for the package name
- clean up of github logic
- REPLACE_PERL
- updated distinfo with BLAKE2s
Version 1.22
* Fix file header validation
* Fix key verification in sequential access
* Fix testing with DejaGNU 1.6.3
* Fix stack overflow in print_usage
* Fix a leak of avail entry on pushing a new avail block
The leak would occur if the original avail table had odd number of entries.
* New gdbmtool variables: errorexit, errormask, trace, timing
"Errorexit" and "errormask" control which GDBM errors would cause the
program termination and emitting a diagnostic message,
correspondingly. Both variables are comma-delimited lists of error
codes.
The "trace" variable enables tracing of the gdbmtool commands.
The "timing" variable, when set, instructs gdbmtool to print time
spent in each command it runs.
* New gdbmtool options: -t (--trace), and -T (--timing)
3.14.8
Back-out all changes to automatically use RETURNING for `SqliteExtDatabase`,
`CSqliteExtDatabase` and `APSWDatabase`. The issue I found is that when a
RETURNING cursor is not fully-consumed, any parent SAVEPOINT (and possibly
transaction) would not be able to be released. Since this is a
backwards-incompatible change, I am going to back it out for now.
Returning clause can still be specified for Sqlite, however it just needs to be
done so manually rather than having it applied automatically.
3.14.7
Fix bug in APSW extension with Sqlite 3.35 and newer, due to handling of last
insert rowid with RETURNING.
3.14.6
Fix pesky bug in new `last_insert_id()` on the `SqliteExtDatabase`.
3.14.5
This release contains a number of bug-fixes and small improvements.
* Only raise `DoesNotExist` when `lazy_load` is enabled on ForeignKeyField.
* Add missing convenience method `ModelSelect.get_or_none()`
* Allow `ForeignKeyField` to specify a custom `BackrefAccessorClass`,
* Ensure foreign-key-specific conversions are applied on INSERT and UPDATE.
* Add handling of MySQL error 4031 (inactivity timeout) to the `ReconnectMixin`
helper class.
* Support specification of conflict target for ON CONFLICT/DO NOTHING.
* Add `encoding` parameter to the DataSet `freeze()` and `thaw()` methods.
* Fix bug which prevented `DeferredForeignKey` from being used as a model's
primary key.
* Ensure foreign key's related object cache is cleared when the foreign-key is
set to `None`.
* Allow specification of `(schema, table)` to be used with CREATE TABLE AS...,
* Allow reusing open connections with DataSet.
* Add `highlight()` and `snippet()` helpers to Sqlite `SearchField`, for use
with full-text search extension.
* Preserve user-provided aliases in column names.
* Add support for Sqlite 3.37 strict tables.
* Ensure database is inherited when using `ThreadSafeDatabaseMetadata`, and
also adds an implementation in `playhouse.shortcuts` along with basic unit
tests.
* Better handling of Model's dirty fields when saving.
* Add basic support for MariaDB connector driver in `playhouse.mysql_ext`.
* Begin a basic implementation for a psycopg3-compatible pg database.
* Add provisional support for RETURNING when using the appropriate versions of
Sqlite or MariaDB.
All checksums have been double-checked against existing RMD160 and
SHA512 hashes
The following distfiles could not be fetched (some may be only fetched
conditionally):
./databases/cstore/distinfo D6.data.ros.gz
./databases/cstore/distinfo cstore0.2.tar.gz
./databases/cstore/distinfo data4.tar.gz
Upstream changes:
1.59 2021-02-22T13:25:52Z
- Added error detection for when a result set's callback doesn't return
an arrayref or arrayrefs
- Moved to GitLab for issue tracking
1.58 2020-11-02T13:34:48Z
- Added the ability for mock result sets to set custom attributes for
statement handles. Thanks to Erik Huelsmann for testing the new feature.
Upstream changes:
1.05 - Todd Rinaldo <toddr@cpan.org> 2020-12-11
- Additional tests for known corner cases.
- Use Perl_warn not warn in XS.
- Test cleanup. Abstract helper calls.
- Macro cleanup in XS for code readability.
1.04 - Todd Rinaldo <toddr@cpan.org> 2020-12-10
- add utf8 mode for read/write of database
- Major re-factor to remove SVs from the CDB internals. SVs are only generated at return.
This was in attempt to address a known memory leak in the previous implementation.
- CDB_File will warn if undef keys or values are passed to insert. This is meant to be
consistent with how Perl handles the situation.
- Consistently tidy CDB_File.xs and remove tabs
- Make unit tests use File::Temp so we don't leave temp files laying around
- update Devel::PPPort to 3.62
1.03 - Todd Rinaldo <toddr@cpan.org> 2020-11-13
WARNING: due to a bug (#12) on 1.02, CDB databases generated by 1.02 will
not be legible by all other versions and vice-versa. It is recommended you
re-generate any databases generated by that version.
- #12 Fix incompatibility with CDB files due to accidental algorithm change.
Add xt tests for 0.96 databases to assure this does not happen again.
- Fixup Changelog for metacpan
- #11 Add additional benchmark test in git for.
- Provide the CDB_File algorithm in POD in the event the original document is lost.
- Update Devel::PPPort to 3.54
Version 3.12.1 fixes a number of bugs:
- Fixed a bug that caused a multi-document transaction to fail when the first
operation was large bulk write (>48MB) that required splitting a batched
write command (`PYTHON-2915`_).
- Fixed a bug that caused the ``tlsDisableOCSPEndpointCheck`` URI option to
be applied incorrectly (`PYTHON-2866`_).
Changes in MySQL 5.7.36
Security Notes
Bugs Fixed
Security Notes
The linked OpenSSL library for MySQL Server has been updated to version 1.1.1l. Issues fixed in the new OpenSSL version are described at https://www.openssl.org/news/cl111.txt and and http://www.openssl.org/news/vulnerabilities.html.
Bugs Fixed
Incompatible Change: For all SELECT statements on a view, the query digest was based on the view definition. As a result, different queries had the same digest and aggregated together in the Performance Schema table events_statements_summary_by_digest, so statistics in that table were not usable for distinguishing distinct SELECT statements.
The query digest for each SELECT statement on a view now is based on the SELECT, not the view definition. This enables distinguishing distinct SELECT statements in the events_statements_summary_by_digest table. However, tools that use query digests may need some adjustment to account for this change. For example, MySQL Enterprise Firewall and query rewrite plugins rely on query digests and existing rules for them that are associated with views may need to be updated.
InnoDB: With undo log truncation enabled (innodb_undo_log_truncate=ON), it was possible for a deadlock and eventual failure to occur when an undo log truncate operation was initiated after a version upgrade from MySQL 5.6 to MySQL 5.7.34 or earlier. A patch introduced in MySQL 5.7.35
[Note] InnoDB: Found duplicate reference rseg: 33 space: 1 page: 3
[Note] InnoDB: Reset pre-5.7.2 rseg: 1 after duplicate is found.
If pre-5.7.2 rollback segment slots have no undo data to purge, a message similar to the following is emitted:
[Note] InnoDB: Successfully reset 32 pre-5.7.2 rseg slots.
If undo data is found in pre-5.7.2 rollback segment slots, a message similar to the following is emitted recommending a slow shutdown and restart:
[Note] InnoDB: pre-5.7.2 rseg: 2 holds data to be purged.
History length: 1. Recommend slow shutdown with innodb_fast_shutdown=0 and restart
InnoDB: Truncation of an undo tablespace during use by an active transaction raised an assertion failure. The transaction was prematurely marked as complete, permitting the truncation operation.
InnoDB: Deleting or updating a row from a parent table initiated a cascading SET NULL operation on the child table that set a virtual column value to NULL. The virtual column value should have been derived from the base column value.
Thanks to Yin Peng at Tencent for the contribution.
InnoDB: The InnoDB recovery process did not recognize that page compression had been applied to data that was being recovered, causing the tablespace data file to increase in size during the redo log apply phase, which could lead to a recovery failure for systems approaching a disk-full state.
Replication: The error messages issued by MySQL Replication when GTIDs required for auto-positioning have been purged could be incorrectly assigned or scrambled in some situations.
Replication: The contents of the gtid_executed and gtid_purged GTID sets were not persisted after restoring a dump taken using mysqldump. The dump file sequence has now been changed so that the mysql schema (which contains the mysql.gtid_executed table) is not dropped after the gtid_purged GTID set is written. A new option --skip-mysql-schema is added for mysqldump which lets you choose not to drop the mysql schema at all.
JSON: Conversion of JSON values to text caused linear growth of the destination string, resulting in an unnecessarily high number of reallocations. Now this process uses exponential growth instead, to reduce the number of allocations required.
This fix originally appeared in MySQL 8.0 and was backported to MySQL 5.7 by Annirudh Prasad, whom we thank for the contribution.
Concurrent insert operations on multiple tables with full-text indexes caused a large number of full-text index synchronization requests, resulting in an out of memory condition.
When a query uses a temporary table for aggregation, the group by item is used as a unique constraint on the temporary table: If the item value is already present, the row is updated; otherwise, a new row is inserted into the temporary table. If the item has a result field or reference item, it it evaluated twice, once to check whether the result exists in the temporary table and, if not, again while constructing the row to be inserted. When the group by item was nondeterministic, the result value used to check for existence differed from that with which an insert was attempted, causing the insert to be rejected if the value already existed in the table.
We fix this by using the hash of any nondeterministic items as the unique constraint, so that the hash is evaluated once only.
Quote handling was improved for the SHOW GRANTS statement.
OpenLDAP 2.5.8 Release (2021/10/11)
Fixed libldap ldap_int_tls_connect: isdigit() requires unsigned char
Fixed libldap memory leak in ldap_get_option LDAP_OPT_X_TLS_PEERCERT
Fixed slapd to allow normalized values for namingContexts in cn=monitor
Fixed slapd to normalize the suffix in rootDSE
Fixed slapd slapadd to avoid destroying configDB prematurely
Fixed slapd to not spam logs with lastbind information
Fixed slapd slaptest migration to correctly set olcTSLVerifyClient
Fixed slapd-mdb multival delete handling
Fixed slapd-sql ldap_entry_objectclass table for mariadb/mysql
Fixed slapd-wt multiple issues
Fixed slapd-wt to close cache db correctly
Fixed slapo-ppolicy to restore OpenLDAP 2.4 compatibilty
Fixed slapo-syncprov to free uuid list when finished replaying sessionlog
Build
Fixed libldap result.c compilation on musl systems
Fixed slapd duplicate definition of peerbv
Fixed test suite with memberof modular builds
Contrib
Added man page for ppm contrib module
Fix crash when pwdCheckModuleArg is not defined for ppm
Documentation
Fixed guide download link for heimdal
Fixed guide documentation for TLSECName
Fixed guide documentation missing tags
Fixed guide loadbalancer typo
Fixed guide synprov-nopresent redundant text
Fixed guide various typos and fix config alignment
Removed ppolicy.schema from servers/slapd/schema/README
Fixed slapd.conf(5)/slapd-config(5) to document default for database monitoring
Fixed slapd-meta(5)/slapd-asyncmeta(5) verbiage for try-propagate
Fixed slapo-syncprov(5) to note entryCSN indexing is highly recommended
repmgr 5.3.0 is a major release.
This release provides support for PostgreSQL 14, released in September 2021.
A.1.1. Improvements
repmgr standby switchover: Improve handling of node rejoin failure on the demotion candidate.
Previously repmgr did not check whether repmgr node rejoin actually succeeded on the demotion candidate, and would always wait up to node_rejoin_timeout seconds for it to attach to the promotion candidate, even if this would never happen.
This makes it easier to identify unexpected events during a switchover operation, such as the demotion candidate being unexpectedly restarted by an external process.
Note that the output of the repmgr node rejoin operation on the demotion candidate will now be logged to a temporary file on that node; the location of the file will be reported in the error message, if one is emitted.
repmgrd: at startup, if node record is marked as "inactive", attempt to set it to "active".
This behaviour can be overridden by setting the configuration parameter repmgrd_exit_on_inactive_node to true.
repmgr node rejoin: emit rejoin target note information as NOTICE.
This makes it clearer what repmgr is trying to do.
repmgr node check: option --repmgrd added to check repmgrd status.
Add %p event notification parameter providing the node ID of the former primary for the repmgrd_failover_promote event.
A.1.2. Bug fixes
repmgr standby clone: if using --replication-conf-only on a node which was set up without replication slots, but the repmgr configuration was since changed to use_replication_slots=1, repmgr will now set slot_name in the node record, if it was previously empty.
repmgrd: rename internal shared library functions to minimize the risk of clashes with other shared libraries.
This does not affect user-facing SQL functions. However an upgrade of the installed extension version is required.
repmgrd: ensure short option -s is accepted.
5.2.0 (2021-10-03)
=====================
Features
- 1. Added support Python 3.10
2. Started shipping platform-specific wheels with the ``musl`` tag targeting typical Alpine Linux runtimes.
3. Started shipping platform-specific arm64 wheels for Apple Silicon.
Bugfixes
- Fixed pure-python implementation that used to raise "Dictionary changed during iteration" error when iterated view (``.keys()``, ``.values()`` or ``.items()``) was created before the dictionary's content change.
Upstream changes:
0.57 2021-09-27
- `$conn->dsn` and `$conn->driver_name` accessors
- Use of ExtUtils::MakeMaker instead of Module::Build
- Corrected, reduced, properly declared prerequisites
- POD tests in `xt/`
- Simplified SQLite version check
- Doc typo fix. Thanks to Michael R. Davis
PostgreSQL 14 contains many new features and enhancements, including:
Stored procedures can now return data via OUT parameters.
The SQL-standard SEARCH and CYCLE options for common table expressions have been implemented.
Subscripting can now be applied to any data type for which it is a useful notation, not only arrays. In this release, the jsonb and hstore types have gained subscripting operators.
Range types have been extended by adding multiranges, allowing representation of noncontiguous data ranges.
Numerous performance improvements have been made for parallel queries, heavily-concurrent workloads, partitioned tables, logical replication, and vacuuming.
B-tree index updates are managed more efficiently, reducing index bloat.
VACUUM automatically becomes more aggressive, and skips inessential cleanup, if the database starts to approach a transaction ID wraparound condition.
Extended statistics can now be collected on expressions, allowing better planning results for complex queries.
libpq now has the ability to pipeline multiple queries, which can boost throughput over high-latency connections.
Redis 6.2.6
Upgrade urgency: SECURITY, contains fixes to security issues.
Security Fixes:
* (CVE-2021-41099) Integer to heap buffer overflow handling certain string
commands and network payloads, when proto-max-bulk-len is manually configured
to a non-default, very large value [reported by yiyuaner].
* (CVE-2021-32762) Integer to heap buffer overflow issue in redis-cli and
redis-sentinel parsing large multi-bulk replies on some older and less common
platforms [reported by Microsoft Vulnerability Research].
* (CVE-2021-32687) Integer to heap buffer overflow with intsets, when
set-max-intset-entries is manually configured to a non-default, very large
value [reported by Pawel Wieczorkiewicz, AWS].
* (CVE-2021-32675) Denial Of Service when processing RESP request payloads with
a large number of elements on many connections.
* (CVE-2021-32672) Random heap reading issue with Lua Debugger [reported by
Meir Shpilraien].
* (CVE-2021-32628) Integer to heap buffer overflow handling ziplist-encoded
data types, when configuring a large, non-default value for
hash-max-ziplist-entries, hash-max-ziplist-value, zset-max-ziplist-entries
or zset-max-ziplist-value [reported by sundb].
* (CVE-2021-32627) Integer to heap buffer overflow issue with streams, when
configuring a non-default, large value for proto-max-bulk-len and
client-query-buffer-limit [reported by sundb].
* (CVE-2021-32626) Specially crafted Lua scripts may result with Heap buffer
overflow [reported by Meir Shpilraien].
Bug fixes that involve behavior changes:
* GEO* STORE with empty source key deletes the destination key and return 0
Previously it would have returned an empty array like the non-STORE variant.
* PUBSUB NUMPAT replies with number of patterns rather than number of subscriptions
This actually changed in 6.2.0 but was overlooked and omitted from the release notes.
Bug fixes that are only applicable to previous releases of Redis 6.2:
* Fix CLIENT PAUSE, used an old timeout from previous PAUSE
* Fix CLIENT PAUSE in a replica would mess the replication offset
* Add some missing error statistics in INFO errorstats
Other bug fixes:
* Fix incorrect reply of COMMAND command key positions for MIGRATE command
* Fix appendfsync to always guarantee fsync before reply, on MacOS and FreeBSD (kqueue)
* Fix the wrong mis-detection of sync_file_range system call, affecting performance
CLI tools:
* When redis-cli received ASK response, it didn't handle it
Improvements:
* Add latency monitor sample when key is deleted via lazy expire
* Sanitize corrupt payload improvements
* Delete empty keys when loading RDB file or handling a RESTORE command
(beta) Support for downsampling using continuous aggregate
(Experimental) Support for storage of exemplars
Performance improvements
Support of Thanos Store API
Build 127 (2020-08-22)
Bug fixes
- It was no longer possible to run GRANT or REVOKE statements.
Build 126 (2020-08-09)
Enhancements
- WbExport will now abort if a text export is started without a quote character but a quote character would be needed.
- For PostgreSQL, subscriptions and publications for logical replication are now shown in the DbTree and DbExplorer.
- For PostgreSQL, the attribute of IDENTITY REPLICA is now shown in the generated DDL.
- "Make uppercase" and "Make lowercase" now also work if no text is selected. In that case, (only) the character at the cursor is changed
- Passwords can now securely be encrypted using a master password. The old password encryption is no longer supported.
- The properties base profile storage (wb-profiles.properties) is now the default. The old XML format is still supported.
- WbVarDef now supports conditional execution.
- WbCopy now supports views as the source by using -includeViews=true.
- The result panel is now automatically displayed if it was minimized before running a statement.
- It's now possible to configure italic and bold for syntax highlighting
- For SQL Server, partitioning information about a table is now included in the generated DDL
- WbImport can now import all sheets of a spreadsheet into the same table when -sheetName=* is specified together with -table
- The annotation @WbResultAsText can now be used to turn the "text mode" display on for multiple statements
- For PostgreSQL, the "match mode" of foreign key constraints is now displayed and included in the generated DDL
- A new Windows launcher is available that removes the need for a separate 64 and 32 bit executables and can optionally download a JRE
- It's now possible to optionally open the saved file when using "Save As"
- It's now possible to use a special variable in WbImport to assign the input file's line number to a column.
- It's now possible to cycle through the result tabs using a keyboard shortcut.
- When exporting a single table from the table list (Database Explorer or DbTree) it's now possible to select the columns to export.
- It's now possible to display results in GUI mode as "text" in the messages area.
- It's now possible to define a SQL expression to be used in the INSERT statement that WbImport uses to apply SQL functions on the input value.
- Selecting foreign key values is now also supported for DELETE statements.
- The list of shortcut keys can now be exported (with the usual "Save As" formats)
- Invoking code completion in the WHERE clause without a column now displays all columns of all used tables of the query
- For MySQL and MariaDB versions that support them, check constraints are now shown in the generated DDL
- Recently opened files are now accessible through the "File" menu.
- For PostgreSQL, default values on columns of views are now shown in the generated DDL
- For HSQLDB 2.5, the new options for temporal tables are now shown in the generated DDL
- Added new command WbSetSchema to change the current schema using the driver's JDBC API
- For PostgreSQL 12, the new computed columns are now shown in the generated DDL
- The tooltip of a column header now always includes the table name if available
Bug fixes
- For Redshift, tables sources were not always displayed
- When using WbImport with mode=upsert and a table that has a multi-column primary key, the wrong MERGE statement was generated for SQL Server, HSQLDB and DB2
- When exporting to ODS, the generated files could not be opened with Microsoft Excel.
- Using nested WbIncludes failed with an exception
- When invoking code-completion in the ON part of a JOIN, not tables where shown.
- When selecting a global SSH configuration for a connection profile, it was no longer possible to revert to a "non-global" configuration
- When specifying a SSH Tunnel without a password and no keyfile, SQL Workbench did not prompt for the password
- For WbCopy, when using -createTarget=true together with -targetSchema the new table was created in the wrong schema
- Fixed an issue when loading the SQL Server JDBC driver with certain versions of OpenJDK
- For SQL Server, when including the CREATE TABLE statement in the output when exporting data as SQL the export failed
- WbDataDiff did not work properly if the key columns contained NULL values.
- When exporting or saving data, it was not possible to enter the filename manually into the dialog.
- Month names were case sensitive when using -dateFormat or -timestampFormat with WbImport
- For Oracle 18c and later, index information wasn't displayed correctly.
- For Oracle 18c and later, DDL for identity columns wasn't generated correctly.
- For Oracle, the generated IGNORE_DUP_KEY hint was incorrect if the order of the PK columns was different to the column order of the table.
- Assigning multiple variables with WbVarDef was not possible if -variable was used.
- For Redshift, finding tables on the search path was no longer working.
- For PostgreSQL, using the JDBC driver 42.2.11, the value for columns defined as "timestamp with time zone" could not be retrieved any longer.
- The cursor position wasn't calculated correctly, if real tabs where used in the editor (regression from 125.6)
- Sorting columns defined as arrays in the result did not work properly.
- The new Windows launcher (introduced in 125.5) did not work if run from a directory with spaces in the name
- Fixed some problems with the cursor position in the editor with High-DPI displays (on Windows)
- For PostgreSQL, the DDL for partitioned tables with sub-partitions was not generated correctly
- Using WbImport with temp tables as the target did not work with PostgreSQL and SQL Server
- When using UCanAccess certain types of "linked columns" are now displayed correctly.
- For DB2 on AS/400 the CSID attribute was also generated for data types where it wasn't relevant.
- Individual statements were not detected correctly in the editor if the text started with a semicolon.
- "Create SQL List" did not work properly if only a part of the first or last line was selected.
- For PostgreSQL, the dependency display of trigger functions wasn't working any more.
- When using Page Down when the editor text was shorter than the number of lines, the display was corrupted.
- When closing the macro dialog while macros were filtered, only the visible macros were saved.
- For PostgreSQL, check constraints and default values were no longer displayed correctly in the generated DDL
- For XLSX exports, using -infoSheet did not work any longer
- When using -endRow with WbImport, the following rows weren't exported, but the input file was still read until the end
- Filtering the list of macros did no longer work correctly.
- For SQLite, all values are now retrieved as strings to workaround problems with the lack of explicit data types.
- Common table expressions containing DML are now correctly identified when the "Read Only" option is selected in the connection profile.
- The XML file generated by WbSchemaDiff was incorrect when multiple views need to be dropped.
- The SQL formatter removed trailing semicolons if only a single statement was formatted.
- Fractional seconds were not correcly formatted in timestamp literals in generated SQL statements.
- When editing a result with columns from multiple tables where the columns were not fully qualified, an exception occurred rather than a usable error message.
- Conditional execution of Workbench commands using -ifDefined and -ifNotDefined behaved the same as -ifEmpty and -ifNotEmpty
- For Postgres, infinity values for timestamp with time zone could not be read from the database.
- If the JDBC driver returned LocalDateTime or LocalDate values, those weren't formatted properly in SQL statements or text files.
Build 125 (2019-05-08)
Enhancements
- All additional libraries are now expected in the directory "ext".
- In the DbExplorer, using "Show Rowcounts", now displays the row counts as a new column in the table list.
- WbDataDiff now also supports -clobAsFile when generating SQL statements
- For WbExport and WbDataDiff -clobFileThreshold has been added
- For MariaDB 10.3, sequences are now displayed in the DbExplorer
- The "Manage Macros" dialog now also supports filtering of the macros.
- For Oracle, the generated DDL for tables now contains the definitions of extended column statistics.
- It's now possible to control the behaviour of WbDefineVar when NULL values are retrieved through a SQL query
- It's now possible to specify a message with WbEcho that is displayed even if verbose logging was turned off (e.g. using -verbose=false for WbInclude)
- For WbExport and WbImport, it's now possible to define the values for -quoteChar and -delimiter using unicode escape notation (e.g. \0008)
- WbDefineVar now supports the -silent option so suppress the details about the defined variable
- For conditional execution of some SQL Workbench/J commands, checking for the (non) existence of files or tables is now supported
- For Postgres, comments on foreign key constraints are now displayed in the DbExplorer and the generated DDL source
- Confirmation to close "Other tabs" can now be configured separately.
Bug fixes
- For SQL Server, DateTime values were not displayed correctly
- For Oracle the source of materialized views using ON PREBUILT TABLE was incorrect.
- Relative file names were treated differently by the SQL Workbench commands and the ifFileExists check.
- For MySQL, the source of triggers wasn't displayed.
- Copy to clipboard did not use the format string for decimal values
- For Oracle, using -mode=upsert with CLOB columns did not work for WbImport
- For some JDBC drivers, the console did not display the column's alias, but the real column name in a result
- For Postgres, nesting dollar quoted strings did not work.
- When retrieving XML columns using a driver that supports the JDBC API getXML() NULL values would result in an error.
- For Postgres, the generated source for an extension in the DbExplorer was not correct.
- The detection of the DB2 version was no long working correctly
- Numbers that can't be stored in Excel's file format are now exported as strings rather than numbers
- For Oracle, certain timestamp values were incorrectly exported.
- For Oracle 12, statements using "WITH FUNCTION" or "/*+ WITH_PLSQL */" to create inline PL/SQL functions could not be executed.
- WbCall did not display values for OUT parameters for all data types.
- For Oracle, the status of foreign keys (enabled, validated) was no longer displayed in the DbExplorer.
- For PostgreSQL, retrieving the policies for row level security was not working for versions prior to 10 any more.
- When using -profileStorage wih a filename without a path, files located in the config directory where not found
- Generating a SQL script using WbDataDiff on tables with timestamptz columns in Postgres did not work
5.48.0 (2021-09-01)
* Make the unused_associations plugin association reflection tracking work
correctly when combining coverage runs (jeremyevans)
* Add Database#like_without_collate on MSSQL, to avoid using COLLATE on LIKE
arguments, which can significantly improve performance (jeremyevans)
* Add Model::Errors#full_message private method for easiest i18n support for
errors with multiple attributes (jeremyevans) (#1779)
Cross-platform TUI database management tool written in Rust.
Features:
- Cross-platform support (macOS, Windows, Linux)
- Multiple Database support (MySQL, PostgreSQL, SQLite)
- Intuitive keyboard only control
As of MySQL 5.7, installations created using --initialize no longer
have an empty root password; one is generated as part of the process
and marked expired. (This message is in part simply advice to read the
documentation. However, it's possible other downstream packagers use
--initialize-insecure instead, which retains the old empty password
approach, so this could be a pkgsrc-specific detail to consider.)
Version 1.21, 2021-09-02
* Crash tolerance
By default it is possible for an abrupt crash (e.g., power failure,
OS kernel panic, or application process crash) to corrupt the gdbm
database file. A new Linux-only mechanism enables applications to
recover the database state corresponding to the most recent
successful gdbm_sync() call before the crash. See the chapter 17
"Crash Tolerance" in the GDBM manual.
* New database file format: numsync
The new "numsync" database format is designed to better support
crash tolerance. To create a database in numsync format, the gdbm_open
(or gdbm_fd_open) function must be given the GDBM_NEWDB|GDBM_NUMSYNC
flags. The GDBM_NUMSYNC flag also takes effect when used together
with GDBM_WRCREAT, provided that the new file is created.
New function gdbm_convert() is provided for converting the databases
from standard GDBM format to numsync and vice versa.
The gdbmtool tool can also be used for converting databases between
these two formats.
* Changes in gdbmtool
** Fix string output in non-ASCII encodings
Printable multi-byte sequences are correctly represented on output.
This also fixes octal representation of unprintable characters.
** The filename variable
This variable supplies the name of database file for use in "open"
command, if the latter is called without arguments. If "open" is
called with the file name argument, the "filename" variable is
initialized to this value.
** The fd variable
If set, its value must be an open file descriptor referring to a
GDBM database file. The "open" command will use gdbm_fd_open
function to use this file. Upon closing the database, this
descriptor will be closed and the variable will be unset.
The file descriptor to use can also be supplied using the
-d (--db-descriptor) command line option.
** The format variable
Defines the format in which new databases will be created. Allowed
values are: "standard" (default) and "numsync".
** New commands: upgrade and downgrade
The "upgrade" command converts current database to the numsync
(extended) format. The "downgrade" command converts current database
to the standard format.
** New command: snapshot
The "snapshot" command is part of the new crash tolerance support.
Given the names of two snapshot files, it analyzes them and selects
the one to be used for database recovery. See the GDBM manual,
section 17.5 "Manual crash recovery" for a detailed discussion of its
use.
