Active Record -- Object-relational mapping in Rails
Active Record connects classes to relational database tables to establish an
almost zero-configuration persistence layer for applications. The library
provides a base class that, when subclassed, sets up a mapping between the
new class and an existing table in the database. In the context of an
application, these classes are commonly referred to as *models*. Models can
also be connected to other models; this is done by defining *associations*.
Active Record relies heavily on naming in that it uses class and association
names to establish mappings between respective database tables and foreign
key columns. Although these mappings can be defined explicitly, it's
recommended to follow naming conventions, especially when getting started
with the library.
This is for Ruby on Rails 7.0.
Please consult the Solr Upgrade notes for details and links specified below:
https://github.com/apache/solr/blob/main/solr/solr-ref-guide/src/solr-upgrade-notes.adoc
====================================================================
Changelog:
Solr 8.11
See the 8.11 Release Notes for an overview of the main new features of Solr 8.11.
When upgrading to 8.11.x users should be aware of the following major changes from 8.10.
Support for Multiple Authentication Schemes
Two new authentication and authorization plugins provide support for configuring multiple authentication schemes.
The MultiAuthPlugin allows combining two or more authentication approaches, such as JWT and Basic authentication.
The MultiAuthRuleBasedAuthorizationPlugin is used when the MultiAuthPlugin is also in use, and combines the various roles defined for all plugins to determine the proper role assignment for the user account.
For information on configuring these plugins, see the following sections:
Combining Basic Authentication with Other Schemes
Multiple Authorization Plugins
ZooKeeper chroot
It’s now possible to create the ZooKeeper chroot at startup if it does not already exist. See the section Using the -z Parameter with bin/solr for an example.
Other Changes
A few other minor changes are worth noting:
The config-read pre-defined permission now correctly governs access for various configuration-related APIs. See also Predefined Permissions.
The S3BackupRepository supports configuring the AWS Profile, if necessary. See also S3BackupRepository.
Additionally, backups will now properly succeed after SPLITSHARD operations, and will correctly handle incremental backup purges.
SolrJ now supports uploading configsets.
Solr 8.10
See the 8.10 Release Notes for an overview of the main new features of Solr 8.10.
When upgrading to 8.10.x users should be aware of the following major changes from 8.9.
Schema Designer UI
A new screen has been added to the Admin UI that allows you to interactively design a Solr schema using your documents.
The designer screen provides a safe environment for you to:
Upload or paste sample documents to identify fields.
Get a "first" guess at what Solr thinks the field types in the fields should be.
Edit fields, field types, dynamic fields, and supporting files.
See how a field’s analysis will impact your text.
Test how schema changes will impact query-time behavior.
Save your changes to a configset to use with a new collection.
See the section Schema Designer for full details.
Backups in S3
Following the redesign of backups in Solr 8.8 that allowed storage of incremental backups in Google Cloud environments, Solr 8.10 provides support for storing backups in Amazon S3 buckets.
See the section S3BackupRepository for how to configure.
Security Admin UI
Solr’s Admin UI also got a new screen to support management of users, roles, and permissions.
The new UI works when authentication and/or authorization has been enabled with bin/solr auth or by manually installing a security.json file. Before this, it provides a warning that your Solr instance is unsecured.
See the section Security UI for details.
Solr SQL Improvements
A number of improvements have been made in Solr’s SQL functionality:
Support added for LIKE, IS NOT NULL, IS NULL, and wildcards (for simplistic LIKE functionality).
Two new aggregation functions, COUNT(DISTINCT field) and APPROX_COUNT_DISTINCT(field), have been added.
Queries using an ORDER BY clause can support OFFSET and FETCH operations.
Multi-valued fields can now be returned.
User permissions have been simplified so access to query endpoints /sql, /select, and /export is sufficient for full access for all SQL queries.
shards.preference
A new option for the shards.preference parameter allows selection of nodes based on whether or not the replica is a leader. Now adding shards.preference=replica.leader:false will limit queries only to replicas which are not currently their shard’s leader.
See the section shards.preference Parameter for details and examples.
Metrics & Prometheus Exporter
A new expr option in the Metrics API allows for more advanced filtering of metrics based on regular expressions. See the section Metrics API for examples.
The Prometheus Exporter’s default solr-exporter.config has been improved to use the new expr option in the Metrics API to get a smaller set of metrics. The default metrics exported still include most metrics, but the configuration will be easier to trim as needed. This should help provide performance improvements in busy clusters being monitored by Prometheus.
ZooKeeper Credentials
ZooKeeper credentials can now be stored in a file whose location is defined with a system property instead of being passed in plain-text. See Out of the Box Credential Implementations for how to set this up.
Solr 8.9
See the 8.9 Release Notes for an overview of the main new features of Solr 8.9.
When upgrading to 8.9.x users should be aware of the following major changes from 8.8.
Backup and Restore
Solr 8.9 introduces extensive changes to Solr’s backup and restore support.
A new backup format has been introduced in Solr 8.9 which replaces the previous snapshot-based backup. This new format enables ‘incremental’ backups. Repeated backups to a given location will take advantage of the data stored by their predecessors and will only operate on files that have changed since the previous backup. This is supported by default, simply by storing each backup file in the same location.
The old and new formats are not compatible, although backups in the old format, a full snapshot of all files, can still be used to restore to Solr for the time-being. The old format is officially deprecated, and support for it is likely to be removed in Solr 9.0.
For the time-being the old format can be created by defining a parameter incremental=false. Again, though, this support is likely to be removed in Solr 9.0.
More documentation on backups is available at Backup and Restore.
New Collections API commands for backups:
LISTBACKUP: Lists information about each backup stored at the specified repository location. See List Backups for more details.
DELETEBACKUP: Deletes specified backups from the repository. See Delete Backups for more details.
A new option for backup repository is also available in 8.9, which is to use Google Cloud Storage (GCS). This is a contrib (located in contrib/gcs-repository). See GCSBackupRepository for configuration details. The Solr community is working to add support for S3 buckets in the near future.
Nested Docs
Child Doc Transformer’s childFilter parameter no longer applies query syntax escaping because it’s inconsistent with the rest of Solr and was limiting. This refers to [child childFilter='field:value']. There was no escaping here prior to 8.0 either.
Collapse and Expand
BlockCollapse: If documents have been (or could be) indexed in a way where documents with the same collapse key have been indexed contiguously in the index, a new "block collapse" provides a significant speed improvement over traditional collapse.
See Block Collapsing for details.
Expand Null Groups: A new parameter expand.nullGroup allows an expanded group to be returned containing document with no value in the expanded field. See Expand Component for details.
In-Place Updates
A new request parameter update.partial.requireInPlace=true allows telling Solr to "fail fast" if all of the necessary conditions are not satisfied to allow an in-place update to succeed. See also In-Place Updates.
Metrics History
The Metrics History feature, which allowed long-term storage and aggregation of Solr’s metrics, has been deprecated and will be removed in 9.0.
Embedded Solr Server
When using EmbeddedSolrServer, it will no longer close CoreContainer instances that were passed to it.
Solr 8.8
When upgrading to 8.8.x users should be aware of the following major changes from 8.7.
Nested Documents
When doing atomic/partial updates to a child document:
Supply the _root_ field (the ID of the root document) so that Solr understands you are manipulating a child document and not a root document. In its absence, Solr looks at the _route_ parameter but this may change in the future because it’s not an ideal substitute. If neither are present, Solr assumes you are updating a root document. If this assumption is false, Solr will do a cheap check that usually detects the problem and will throw an exception to alert you of the need to specify the Root ID. This backwards incompatible change was done to increase performance and robustness.
This feature no longer requires stored=true or docValues=true on the _root_ field. You might have it for other purposes though (e.g., for uniqueBlock(…)).
This feature no longer requires the _nest_path_ field, although you probably ought to continue to define it as it’s useful for other things.
Removed Contribs
The search results clustering contrib (Carrot2) has been removed from 8.x Solr due to lack of Java 1.8 compatibility in the dependency that provides online clustering of search results. The contrib will be re-introduced in Solr 9.0.
Learning to Rank
Interleaving support has been added to Learning to Rank (LTR). Currently only the Team Draft Interleaving algorithm is supported. For examples using this feature, see the section Running a Rerank Query Interleaving Two Models.
Metrics
Two metrics have been added for SolrCloud’s Overseer:
solr_metrics_overseer_stateUpdateQueueSize
solr_metrics_overseer_collectionWorkQueueSize
Prometheus Exporter
The ./bin scripts included with the Prometheus Exporter contrib now allow use of custom java options with environment variables. See the section Environment Variable Options for more details.
The default Grafana dashboards now include panels for query performance monitoring. The default Prometheus Exporter configuration includes metrics like queries-per-second (QPS) and 95th percentiles (P95) to populate the new panels.
The default Prometheus Exporter configuration also includes the two new metrics mentioned in the Metrics above.
Solr Home
The internal logic for identifying 'Solr Home' (solr.solr.home) has been refactored to make testing less error prone. Plugin developers using SolrPaths.locateSolrHome() or 'new `SolrResourceLoader’ should check deprecation warnings as existing some existing functionality will be removed in 9.0. SOLR-14934 has more technical details about this change for those concerned.
base_url removed from stored state
As of Solr 8.8.0, the base_url property was removed from the stored state for replicas (SOLR-12182). If you’re able to upgrade SolrJ to 8.8.x for all of your client applications, then you can set -Dsolr.storeBaseUrl=false (introduced in Solr 8.8.1) to better align the stored state in ZooKeeper with future versions of Solr. However, if you are not able to upgrade SolrJ to 8.8.x for all client applications, then leave the default -Dsolr.storeBaseUrl=true so that Solr will continue to store the base_url in ZooKeeper.
You may also see some NPE in collection state updates during a rolling upgrade to 8.8.0 from a previous version of Solr. After upgrading all nodes in your cluster to 8.8.0, collections should fully recover. Trigger another rolling restart if there are any replicas that do not recover after the upgrade to re-elect leaders.
Solr 8.7
See the 8.7 Release Notes for an overview of the main new features of Solr 8.7.
When upgrading to 8.7.x users should be aware of the following major changes from 8.6.
Autoscaling
If upgrading from 8.6.0, please see the 8.6.1 Upgrade notes below for information on performance degradations introduced in 8.6.0 that require some intervention to resolve. If you are already on 8.6.1 or higher, you can ignore these instructions.
Deprecations
The autoscaling framework is now formally deprecated and will be removed in Solr 9.0. The Solr community is working on pluggable API to replace this functionality, with the goal for it to be ready by the time 9.0 is released. Deprecations include: autoscaling policy, triggers, withCollection support, simulation framework, autoscaling suggestions tab in the UI, autoAddReplicas and UTILIZENODE command.
Similarly, rule-based replica placement strategy has been deprecated and will be replaced in Solr 9.0 by APIs for replica placement and cluster events, with plugin-based implementations.
Support for detecting spinning disks has been removed in LUCENE-9576. Corresponding spins metrics in Solr still exist but now they always return false and will be removed in Solr 9.0.
User-Managed Cluster Terminology Updated
Solr has replaced the terms "master" and "slave" in the codebase and all documentation with "leader" and "follower".
This functionality has only changed in terms of parameter names changed, and we do not expect any back-compatibility issues on upgrade to 8.7 or even 9.0 later.
However, users should update their solrconfig.xml files after completing the upgrade on all nodes of a cluster. Comparing your configuration to the updated configuration examples in Index Replication will show examples of what needs to change, but here are the main changes:
On the replication leader, in the definition of the /replication request handler:
Replace "master" with "leader".
Replace "slave" with "follower" if the former term is used in the name of any follower solrconfig.xml file definitions. This file can be named anything, so you can change it to whatever you’d like to call it if you’d like.
Replace "slave" with "follower" if the former term is used in a replication repeater configuration.
On the replication follower, in the definition of the /replication request handler:
Replace "masterUrl" with "leaderUrl".
Replace "slave" with "follower" if the former term is used in a repeater configuration.
JSON Facets
Performance enhancements for the relatedness() statistics function are included with 8.7. These yield the highest benefits with high-cardinality fields and can be disabled if working with lower cardinality fields with a new sweep_collection parameter. See the section relatedness() Options for details.
solr.in.sh / solr.in.cmd
Solr has relied on the SOLR_STOP_WAIT parameter defined in solr.in.sh or solr.in.cmd to determine how long to wait on startup. A new parameter SOLR_START_WAIT allows defining how long Solr should wait for start up to complete.
If the time set by this parameter is exceeded, Solr will exit the startup process and return the last few lines of the solr.log file to the terminal.
By default, this parameter is set to the same value as SOLR_STOP_WAIT.
The default ZooKeeper client timeout (ZK_CLIENT_TIMEOUT) is now 30 seconds (30000 milliseconds) instead of 15.
Configsets
It’s now possible to overwrite an existing configset when uploading changes by supplying the overwrite=true parameter to the Configset API.
A related parameter is cleanup=true, which allows deleting any files from the old configset that are left behind after the overwrite.
The default for both of these parameters is false.
When deleting a collection that has an automatically created configset (i.e., the configset was copied from the _default collection when the collection was created), the configset will also be deleted if it is not in use by any other collection.
Logging
A request ID (rid) is now logged for all distributed search requests (in SolrCloud) which can be used to correlate query events across the system. A parameter disableRequestId=true can be added to disable this if desired.
Solr 8.6.1
See the 8.6.1 Release Notes for an overview of the fixes included in Solr 8.6.1.
When upgrading to 8.6.1 users should be aware of the following major changes from 8.6.0.
Autoscaling
As mentioned in the 8.6 upgrade notes, a default autoscaling policy was provided starting in 8.6.0. This default autoscaling policy resulted in increasingly slow collection creation calls in large clusters (50+ collections).
In 8.6.1 the default autoscaling policy has been removed, and clusters will not use autoscaling unless a policy has explicitly been created. If your cluster is running 8.6.0 and not using an explicit autoscaling policy, upgrade to 8.6.1 and remove the default cluster policy and preferences via the following command.
Replace localhost:8983 with your Solr endpoint.
curl -X POST -H 'Content-type:application/json' -d '{set-cluster-policy : [], set-cluster-preferences : []}' http://localhost:8983/api/cluster/autoscaling
This information is only relevant for users upgrading from 8.6.0. If upgrading from an earlier version to 8.6.1+, this warning can be ignored.
Solr 8.6
See the 8.6 Release Notes for an overview of the main new features of Solr 8.6.
When upgrading to 8.6.x users should be aware of the following major changes from 8.5.
Support for Block-Max WAND
Lucene added support for Block-Max WAND in 8.0, and 8.6 makes this available for Solr also.
This can provide significant performance enhancements by not calculating the score for results which are not likely to appear in the top set of results.
It is enabled when using a new query parameter minExactCount. This parameter tells Solr to accurately count the number of hits accurately until at least this value. Once this value is reached, Solr can skip over documents that don’t have a score high enough to be in the top set of documents, which has the potential for greatly speeding up searches.
It’s important to note that when using this parameter, the hit count of searches may not be accurate. It is guaranteed that the hit count is accurate up to the value of minExactCount, but any returned hit count higher than that may be an approximation.
A new boolean attribute numFoundExact is included in all responses to indicate if the hit count in the response is expected to be exact or not.
More information about this new feature is available in the section minExactCount Parameter.
Autoscaling
NOTE: The default autoscaling policy has been removed as of 8.6.1
Solr now includes a default autoscaling policy. This policy can be overridden with your custom rules or by specifying an empty policy to replace the default.
The ComputePlan action now supports a collection selector to identify collections based on collection properties to determine which collections should be operated on.
Security
Prior to Solr 8.6 Solr APIs which take a file system location, such as core creation, backup, restore, and others, did not validate the path and Solr would allow any absolute or relative path. Starting in 8.6 only paths that are relative to SOLR_HOME, SOLR_DATA_HOME and coreRootDir are allowed by default.
If you need to create a core or store a backup outside the default paths, you will need to tell Solr which paths to allow. A new element in solr.xml called allowPaths takes a comma-separated list of allowed paths.
When using the solr.xml file that ships with 8.6, you can configure the list of paths to allow through the system property solr.allowPaths. Please see bin/solr.in.sh or bin\solr.in.cmd for example usage. Using the value * will allow any path as in earlier versions.
For more on this, see the section Solr.xml Parameters.
Windows SMB shares on the UNC format, such as \\myhost\myshare\mypath are now always disallowed. Please use drive letter mounts instead, i.e., S:\mypath.
A new authorization plugin ExternalRoleRuleBasedAuthorizationPlugin is now available. This plugin allows an authentication plugin (such as JWT) to supply a user’s roles instead of maintaining a user-to-role mapping inside Solr.
When authentication is enabled, the Admin UI Dashboard (main screen) now includes a panel that shows the authentication and authorization plugins in use, the logged in username, and the roles assigned to this user. A new link will also appear in the left-hand navigation to allow a user to log out.
Streaming Expressions
The /export handler now supports streaming expressions to allow limiting the output of the export to only matching documents.
For more information about how to use this, see the section Specifying the Local Streaming Expression.
The stats, facet, and timeseries expressions now support percentiles and standard deviation aggregations.
Highlighting
For the Unified Highlighter: The setting hl.fragsizeIsMinimum now defaults to false because true was found to be a significant performance regression when highlighting lots of text. This will yield longer highlights on average compared to Solr 8.5 but relatively unchanged compared to previous releases. Furthermore, if your application highlights lots of text, you may want to experiment with lowering hl.fragAlignRatio to trade ideal fragment alignment for better performance.
Deprecations
A primary focus of the community is improving Solr’s stability and supportability. With the addition of the package manager system in 8.4, we now have the ability to move some features into plugins maintained by third-parties with the expertise to properly develop and support them. Our goal is to make running Solr easier and less prone to outages and other headaches. In this spirit, the following features have been deprecated and are slated to be removed in Solr 9.0.
Cross Data Center Replication (CDCR), in its current form, is deprecated and is scheduled to be removed in 9.0. This feature is unlikely to be replaced by an identical plugin. However, the community is working on figuring out a replacement feature for disaster recovery and failover.
The Data Import Handler (DIH) is deprecated and is scheduled to be removed in 9.0. Work to replace DIH with a community-supported plugin is underway and may be available soon.
Support to store indexes and backups in HDFS is deprecated and is scheduled to be removed in 9.0. A community-supported version of this may be available as a plugin in the future. For more details, please see SOLR-14021.
Users interested in maintaining a feature as a plugin are encouraged to join the developer mailing list to find out more about how to help.
See NEWS in sources for details. A summary of upstream NEWS:
* Enhancements and bugfixes
PostGIS 3.2.0 (Olivier Courtin Edition)
2021/12/17
This version requires PostgreSQL 9.6 or higher, GEOS 3.6 or higher
, and Proj 4.9+.
Additional features are enabled if you are running GEOS 3.9+
(and ST_MakeValid enhancements with 3.10+),
Proj 6.1+, and PostgreSQL 14+.
Due to some query performance degradation
with the new fast index build that requires PG14,
we have decided to disable the feature by default
until we get more user testing
as to the true impact of real-world queries.
If you are running PG14+, you can reenable it by doing
ALTER OPERATOR FAMILY gist_geometry_ops_2d USING gist
ADD FUNCTION 11 (geometry)
geometry_gist_sortsupport_2d (internal);
and then reindex your gist indexes.
To revert back to the old index behavior:
ALTER OPERATOR FAMILY gist_geometry_ops_2d using gist
DROP FUNCTION 11 (geometry);
and then reindex your gist indexes.
* Breaking changes *
- #5008, Empty geometries are not reported as being within Infinite
distance by ST_DWithin (Sandro Santilli)
- #4824, Removed `--without-wagyu` build option. Using Wagyu is now mandatory to build with MVT support.
- #4933, topology.GetFaceByPoint will not work with topologies having invalid edge linking.
- #4981, ST_StartPoint support any geometry. No longer returns null for non-linestrings.
- #4149, ST_AsMVTGeom now preserves more of original geometry's details at scale close to target extent.
If you need previous simplifying behaviour, you can ST_Simplify the geometry in advance.
(Darafei Praliaskouski)
- Proj 4.9 or higher is required
- #5000, Turn off Window support in ST_AsMVT aggregate
as no real use-case for it and it crashes with random input
(Paul Ramsey)
* New features*
- #4923, topology.ValidateTopologyRelation (Sandro Santilli)
- #4933, topology.GetFaceContainingPoint (Sandro Santilli)
- #2175, ST_Scroll (Sandro Santilli)
- #4841, FindTopology to quickly get a topology record (Sandro Santilli)
- #4869, FindLayer to quickly get a layer record (Sandro Santilli)
- #4851, TopoGeom_addTopoGeom function (Sandro Santilli)
- ST_MakeValid(geometry, options) allows alternative validity building
algorithms with GEOS 3.10 (Paul Ramsey)
- ST_InterpolateRaster() fills in raster cells between sample points
using one of a number of algorithms (inverse weighted distance, average, etc)
using algorithms from GDAL
(Paul Ramsey)
- ST_Contour() generates contour lines from raster values
using algorithms from GDAL (Paul Ramsey)
- ST_SetZ()/ST_SetM() fills in z/m coordinates of a geometry using data read
from a raster (Paul Ramsey)
- New postgis.gdal_vsi_options GUC allows out-db rasters on VSI network
services to be accessed with authentication keys, etc. (Paul Ramsey)
- ST_DumpSegments returns a set of segments of input geometry (Aliaksandr Kalenik)
- #4859, ST_Point, ST_PointZ, ST_PointM, ST_PointZM, constructors
with SRID parameter (Paul Ramsey)
- #4808, ST_ClusterKMeans now supports max_radius argument. Use it when you're not sure what is
the number of clusters but you know what the size of clusters should be. (Darafei Praliaskouski)
dbplyr is the database backend for dplyr. It allows you to use
remote database tables as if they are in-memory data frames by
automatically converting dplyr code into SQL.
Breaks build with python 2.7 because the paths are different there.
Still packages with python 3.6, 3.9, 3.10 on NetBSD.
Bump PKGREVISION for PLIST change.
Upstream changes:
0.143 2021-03-28
- fix RT #134147: Failed test 'SQL::Abstract - no limit' / Thanks to Slaven Rezić and Thibault Duponchelle
0.142 2020-11-22
- module adopted by Alexander Becker
- set up GitHub repository. See this repository for detailed changes.
- fix RT #34847: Avoid double quoting of table name. Fix provided by @dekimsey / Daniel Kimsey
- fix RT #50795: SQL::Abstract properties were ignored for select() if no limit was given. Thanks to Gunnar Hansson
- fix RT #73444: Offset argument wasn't handled properly if the value was 0. Thanks to ALEXBYK
- fix RT #62671: Properly handle 0 values in condition.
- remove Build.PL, switch to EUMM
Avoids build failure if monogb 4.x is installed.
Someone can always add a mongodb option to options.mk if they want, but
this is just to ensure reliable builds
- Updated all the live engine tests, aside from Oracle, to test with
unique registry names, so as to avoid conflicts when multiple
instances are being tested at once. Thanks to Slaven Rezić for the
report (#597).
- Removed `local` directory from the distribution, accidentally included
in v1.2.0. Thanks to gregor herrmann for the report (#600).
5.51.0 (2021-12-01)
* Make eager loading via tactical_eager_loading no longer modify objects who
already have a cached value for the association (jeremyevans)
* Make association cloning handle cases where clone association sets
different :class option than cloned association (jeremyevans)
* Make column schema entries on MySQL include an :extra entry for the Extra
column in DESCRIBE output (bschmeck) (#1791)
* Update mock PostgreSQL adapter to default to PostgreSQL 14 instead of
PostgreSQL 9.5 (jeremyevans)
* Support Dataset#with_recursive :search and :cycle options on PostgreSQL
14+ for result ordering and cycle detection (jeremyevans)
* Avoid method redefined verbose mode warnings in lazy_attributes plugin
(jeremyevans)
Version 1.3.0 - 2021-09-14
* Add GROUPS frame to Window
* Add exclude to Window
* Add method for each type of join
* Remove support for Python older than 3.5
* Support Select queries in conditionals and functions
Version 1.2.2 - 2021-05-16
* Escape table and column names containing double quote
* Use alias on the main table used in UPDATE and INSERT
* Add support for Python 3.9
Version 1.2.1 - 2020-10-08
* Fix AttributeError in Select.params with windows
Version 1.2.0 - 2020-10-05
* Add explicit Windows to Select
* Fix missing Windows definitions in nested expressions
Version 1.1.0 - 2020-01-25
* Add ORDER BY clause to aggregate functions
* Add support for Python 3.8
* Add distinct on Select
Version 1.0.0 - 2018-09-30
* Add Flavor filter_ to fallback to case expression
* Allow to use expression in AtTimeZone
* Fix Select query in returning
* Add comparison predicates
* Add COLLATE
kVS: ----------------------------------------------------------------------
SQLite Release 3.37.0 On 2021-11-27
STRICT tables provide a prescriptive style of data type management, for developers who prefer that kind of thing.
When adding columns that contain a CHECK constraint or a generated column containing a NOT NULL constraint, the ALTER TABLE ADD COLUMN now checks new constraints against preexisting rows in the database and will only proceed if no constraints are violated.
Added the PRAGMA table_list statement.
CLI enhancements:
Add the .connection command, allowing the CLI to keep multiple database connections open at the same time.
Add the --safe command-line option that disables dot-commands and SQL statements that might cause side-effects that extend beyond the single database file named on the command-line.
Performance improvements when reading SQL statements that span many lines.
Added the sqlite3_autovacuum_pages() interface.
The sqlite3_deserialize() does not and has never worked for the TEMP database. That limitation is now noted in the documentation.
The query planner now omits ORDER BY clauses on subqueries and views if removing those clauses does not change the semantics of the query.
The generate_series table-valued function extension is modified so that the first parameter ("START") is now required. This is done as a way to demonstrate how to write table-valued functions with required parameters. The legacy behavior is available using the -DZERO_ARGUMENT_GENERATE_SERIES compile-time option.
Added new sqlite3_changes64() and sqlite3_total_changes64() interfaces.
Added the SQLITE_OPEN_EXRESCODE flag option to sqlite3_open_v2().
Use less memory to hold the database schema.
Currently supports dBase III, Clipper, FoxPro, and Visual FoxPro
tables. Text is returned as unicode, and codepage settings in tables
are honored. Memos and Null fields are supported. Documentation
needs work, but author is very responsive to e-mails.
Not supported: index files (but can create tempory non-file indexes),
auto-incrementing fields, and Varchar fields.
- Fixed partitioned search for deployed changes on Oracle and Exasol to
correctly support plans with more than 250 changes. Thanks to @Nicqu
for the report (#521).
- DBI connections to the registry will now be set to trace level one
when a verbosity level of three or higher is passed to Sqitch
(i.e., `sqitch -vvv`). Thanks to @wkoszek for the suggestion (#155).
- Renamed the "master" branch to "main" and updated all relevant
references.
- Fixed the parsing of the Snowflake account name from the target URL
to that it no longer strips out the region and cloud platform parts.
Also deprecated the use of the region, host, and port config and
environment variables. This is to comply with the new account name
format. Thanks to @ldsingh00 for the report (#544).
- The checkout command will now show a usage statement when no branch
name is passed to it, rather than a series of warnings and a Git error.
Thanks to François Beausoleil for the report (#561).
- The checkout command now works when operating on a Sqitch project in
a subdirectory of a Git project. Thanks to François Beausoleil for the
report and suggested fix (#560).
- Fixed a failing bundle test when a top directory was configured in the
user or system configuration. Thanks to @CodingMinds for the spot
(#587).
- Added support to the Exasol engine for the `AUTHMETHOD` query parameter
(to allow Open ID authentication) and the `SSLCERTIFICATE=SSL_VERIFY_NONE`
query parameter to disable SSL verification. Thanks to Torsten Glunde
for the report (#588).
- Fixed "Use of uninitialized value $engine in concatenation" error when
Sqitch is unable to determine the engine when writing a plan file. Only
happens in the rare case that the core.engine value is not set.
- Improved the error message when attempting to update a plan file and no
project pragma is present. Thanks to Laurentiu Diaconu for the report
(#577).
- Fixed the error message when attempting to deploy a change that has
already been deployed to display the name of the change, rather than a
memory address. Thanks to Neil Mayhew for the report (#579).
- Added destination locking, currently implemented for PostgresQL and
MySQL. On starting a deploy or revert, Sqitch attempts to "lock the
destination" using advisory locks, to ensure that only one instance of
Sqitch makes changes to the database at any one time. This complements
the existing locking, which applies as each change is deployed or
reverted, as that pattern led to failures when multiple instances of
Sqitch were working at once. Thanks to Neil Mayhew for the report
(#579).
- Added the --lock-timeout option to the deploy, revert, rebase, and
checkout commands, to allow for shorter or longer times to wait for an
exclusive target database advisory lock for engines that support it.
Defaults to 60 seconds.
- Fixed the behavior of the plan command's `--max-count` option when used
with `--reverse` to show the proper items. Thanks to Adrian Klaver for
the report (#517).
- Fixed an issue that could cause bugs with the `check` command on
Firebird, Exasol, Oracle, and Vertica. Broken since the `check` command
was added in v1.1.0.
v0.25.0
Changes
Improve SSL option compatibility in URIs
Add Pool methods to determine its min, max, current and idle size
Make it possible to specify a statement name in Connection.prepare()
Implement support for multirange types
Fixes
Make sure timeout callbacks always get cleaned up
Update __all__ statements to a simpler form that is better supported by typecheckers
Fix test_timetz_encoding on Python 3.10
Fix a bunch of ResourceWarnings in the test suite
Fix SSLContext deprecation warnings
Fix the description of the database argument to connect()
Fix parsing of IPv6 addresses in the connection URI
Improve diagnostics of invalid executemany() input
PostgreSQL 14.1, 13.5, 12.9, 11.14, 10.19, and 9.6.24
Security Issues
CVE-2021-23214: Server processes unencrypted bytes from man-in-the-middle
Versions Affected: 9.6 - 14. The security team typically does not test unsupported versions, but this problem is quite old.
When the server is configured to use trust authentication with a clientcert requirement or to use cert authentication, a man-in-the-middle attacker can inject arbitrary SQL queries when a connection is first established, despite the use of SSL certificate verification and encryption.
The PostgreSQL project thanks Jacob Champion for reporting this problem.
CVE-2021-23222: libpq processes unencrypted bytes from man-in-the-middle
Versions Affected: 9.6 - 14. The security team typically does not test unsupported versions, but this problem is quite old.
A man-in-the-middle attacker can inject false responses to the client's first few queries, despite the use of SSL certificate verification and encryption.
If more preconditions hold, the attacker can exfiltrate the client's password or other confidential data that might be transmitted early in a session. The attacker must have a way to trick the client's intended server into making the confidential data accessible to the attacker. A known implementation having that property is a PostgreSQL configuration vulnerable to CVE-2021-23214.
As with any exploitation of CVE-2021-23214, the server must be using trust authentication with a clientcert requirement or using cert authentication. To disclose a password, the client must be in possession of a password, which is atypical when using an authentication configuration vulnerable to CVE-2021-23214. The attacker must have some other way to access the server to retrieve the exfiltrated data (a valid, unprivileged login account would be sufficient).
The PostgreSQL project thanks Jacob Champion for reporting this problem.
Bug Fixes and Improvements
This update fixes over 40 bugs that were reported in the last several months. The issues listed below affect PostgreSQL 14. Some of these issues may also affect other supported versions of PostgreSQL.
Some of these fixes include:
Fix physical replication for cases where the primary crashes after shipping a WAL segment that ends with a partial WAL record. When applying this update, update your standby servers before the primary so that they will be ready to handle the fix if the primary happens to crash.
Fix parallel VACUUM so that it will process indexes below the min_parallel_index_scan_size threshold if the table has at least two indexes that are above that size. This problem does not affect autovacuum. If you are affected by this issue, you should reindex any manually-vacuumed tables.
Fix causes of CREATE INDEX CONCURRENTLY and REINDEX CONCURRENTLY writing corrupt indexes. You should reindex any concurrently-built indexes.
Fix for attaching/detaching a partition that could allow certain INSERT/UPDATE queries to misbehave in active sessions.
Fix for creating a new range type with CREATE TYPE that could cause problems for later event triggers or subsequent executions of the CREATE TYPE command.
Fix updates of element fields in arrays of a domain that is a part of a composite.
Disallow the combination of FETCH FIRST WITH TIES and FOR UPDATE SKIP LOCKED.
Fix corner-case loss of precision in the numeric power() function.
Fix restoration of a Portal's snapshot inside a subtransaction, which could lead to a crash. For example, this could occur in PL/pgSQL when a COMMIT is immediately followed by a BEGIN ... EXCEPTION block that performs a query.
Clean up correctly if a transaction fails after exporting its snapshot. This could occur if a replication slot was created then rolled back, and then another replication slot was created in the same session.
Fix for "overflowed-subtransaction" wraparound tracking on standby servers that could lead to performance degradation.
Ensure that prepared transactions are properly accounted for during promotion of a standby server.
Ensure that the correct lock level is used when renaming a table.
Avoid crash when dropping a role that owns objects being dropped concurrently.
Disallow setting huge_pages to on when shared_memory_type is sysv
Fix query type checking in the PL/pgSQL RETURN QUERY.
Several fixes for pg_dump, including the ability to dump non-global default privileges correctly.
Use the CLDR project's data to map Windows time zone names to IANA time zones.
This update also contains tzdata release 2021e for DST law changes in Fiji, Jordan, Palestine, and Samoa, plus historical corrections for Barbados, Cook Islands, Guyana, Niue, Portugal, and Tonga.
Also, the Pacific/Enderbury zone has been renamed to Pacific/Kanton. Also, the following zones have been merged into nearby, more-populous zones whose clocks have agreed with them since 1970: Africa/Accra, America/Atikokan, America/Blanc-Sablon, America/Creston, America/Curacao, America/Nassau, America/Port_of_Spain, Antarctica/DumontDUrville, and Antarctica/Syowa. In all these cases, the previous zone name remains as an alias.
1.7.5
Released: November 11, 2021
bug
[bug] [tests]
Adjustments to the test suite to accommodate for error message changes occurring as of SQLAlchemy 1.4.27.
1.7.4
Released: October 6, 2021
bug
[bug] [regression]
Fixed a regression that prevented the use of post write hooks on python version lower than 3.9
[bug] [environment]
Fixed issue where the MigrationContext.autocommit_block() feature would fail to function when using a SQLAlchemy engine using 2.0 future mode.
1.7.3
Released: September 17, 2021
bug
[bug] [mypy]
Fixed type annotations for the “constraint_name” argument of operations create_primary_key(), create_foreign_key(). Pull request courtesy TilmanK.
1.7.2
Released: September 17, 2021
bug
[bug] [typing]
Added missing attributes from context stubs.
[bug] [mypy]
Fixed an import in one of the .pyi files that was triggering an assertion error in some versions of mypy.
[bug] [ops] [regression]
Fixed issue where registration of custom ops was prone to failure due to the registration process running exec() on generated code that as of the 1.7 series includes pep-484 annotations, which in the case of end user code would result in name resolution errors when the exec occurs. The logic in question has been altered so that the annotations are rendered as forward references so that the exec() can proceed.
1.7.1
Released: August 30, 2021
bug
[bug] [installation]
Corrected “universal wheel” directive in setup.cfg so that building a wheel does not target Python 2. The PyPi files index for 1.7.0 was corrected manually. Pull request courtesy layday.
[bug] [pep484]
Fixed issue in generated .pyi files where default values for Optional arguments were missing, thereby causing mypy to consider them as required.
[bug] [batch] [regression]
Fixed regression in batch mode due to 883 where the “auto” mode of batch would fail to accommodate any additional migration directives beyond encountering an add_column() directive, due to a mis-application of the conditional logic that was added as part of this change, leading to “recreate” mode not being used in cases where it is required for SQLite such as for unique constraints.
1.7.0
Released: August 30, 2021
changed
[changed] [installation]
Alembic 1.7 now supports Python 3.6 and above; support for prior versions including Python 2.7 has been dropped.
[changed] [installation]
Make the python-dateutil library an optional dependency. This library is only required if the timezone option is used in the Alembic configuration. An extra require named tz is available with pip install alembic[tz] to install it.
[changed] [installation]
The dependency on pkg_resources which is part of setuptools has been removed, so there is no longer any runtime dependency on setuptools. The functionality has been replaced with importlib.metadata and importlib.resources which are both part of Python std.lib, or via pypy dependency importlib-metadata for Python version < 3.8 and importlib-resources for Python version < 3.9 (while importlib.resources was added to Python in 3.7, it did not include the “files” API until 3.9).
feature
[feature] [environment]
Enhance version_locations parsing to handle paths containing spaces. The new configuration option version_path_separator specifies the character to use when splitting the version_locations string. The default for new configurations is version_path_separator = os, which will use os.pathsep (e.g., ; on Windows).
[feature] [tests]
Created a “test suite” similar to the one for SQLAlchemy, allowing developers of third-party dialects to test their code against a set of Alembic tests that have been specially selected to exercise back-end database operations. At the time of release, third-party dialects that have adopted the Alembic test suite to verify compatibility include CockroachDB and SAP ASE (Sybase).
[feature] [general]
pep-484 type annotations have been added throughout the library. Additionally, stub .pyi files have been added for the “dynamically” generated Alembic modules alembic.op and alembic.config, which include complete function signatures and docstrings, so that the functions in these namespaces will have both IDE support (vscode, pycharm, etc) as well as support for typing tools like Mypy. The files themselves are statically generated from their source functions within the source tree.
usecase
[usecase] [batch]
Named CHECK constraints are now supported by batch mode, and will automatically be part of the recreated table assuming they are named. They also can be explicitly dropped using op.drop_constraint(). For “unnamed” CHECK constraints, these are still skipped as they cannot be distinguished from the CHECK constraints that are generated by the Boolean and Enum datatypes.
Note that this change may require adjustments to migrations that drop or rename columns which feature an associated named check constraint, such that an additional op.drop_constraint() directive should be added for that named constraint as there will no longer be an associated column for it; for the Boolean and Enum datatypes, an existing_type keyword may be passed to BatchOperations.drop_constraint as well.
See also Changing the Type of Boolean, Enum and other implicit CHECK datatypes
Including CHECK constraints
bug
[bug] [operations]
Fixed regression due to 803 where the .info and .comment attributes of Table would be lost inside of the DropTableOp class, which when “reversed” into a CreateTableOp would then have lost these elements. Pull request courtesy Nicolas CANIART.
[bug] [batch] [sqlite]
Batch “auto” mode will now select for “recreate” if the add_column() operation is used on SQLite, and the column itself meets the criteria for SQLite where ADD COLUMN is not allowed, in this case a functional or parenthesized SQL expression or a Computed (i.e. generated) column.
[bug] [commands]
Re-implemented the python-editor dependency as a small internal function to avoid the need for external dependencies.
[bug] [postgresql]
Fixed issue where usage of the PostgreSQL postgresql_include option within a Operations.create_index() would raise a KeyError, as the additional column(s) need to be added to the table object used by the construct internally. The issue is equivalent to the SQL Server issue fixed in 513. Pull request courtesy Steven Bronson.
1.6.5
Released: May 27, 2021
bug
[bug] [autogenerate]
Fixed issue where dialect-specific keyword arguments within the DropIndex operation directive would not render in the autogenerated Python code. As support was improved for adding dialect specific arguments to directives as part of 803, in particular arguments such as “postgresql_concurrently” which apply to the actual create/drop of the index, support was needed for these to render even in a drop index operation. Pull request courtesy Jet Zhou.
1.6.4
Released: May 24, 2021
bug
[bug] [op directives] [regression]
Fixed regression caused by just fixed 844 that scaled back the filter for unique=True/index=True too far such that these directives no longer worked for the op.create_table() op, this has been fixed.
1.6.3
Released: May 21, 2021
bug
[bug] [autogenerate] [regression]
Fixed 1.6-series regression where UniqueConstraint and to a lesser extent Index objects would be doubled up in the generated model when the unique=True / index=True flags were used.
[bug] [autogenerate]
Fixed a bug where paths defined in post-write hook options would be wrongly escaped in non posix environment (Windows).
[bug] [regression] [versioning]
Fixed regression where a revision file that contained its own down revision as a dependency would cause an endless loop in the traversal logic.
1.6.2
Released: May 6, 2021
bug
[bug] [regression] [versioning]
Fixed additional regression nearly the same as that of 838 just released in 1.6.1 but within a slightly different codepath, where “alembic downgrade head” (or equivalent) would fail instead of iterating no revisions.
1.6.1
Released: May 6, 2021
bug
[bug] [regression] [versioning]
Fixed regression in new revisioning traversal where “alembic downgrade base” would fail if the database itself were clean and unversioned; additionally repairs the case where downgrade would fail if attempting to downgrade to the current head that is already present.
1.6.0
Released: May 3, 2021
feature
[feature] [autogenerate]
Fix the documentation regarding the default command-line argument position of the revision script filename within the post-write hook arguments. Implement a REVISION_SCRIPT_FILENAME token, enabling the position to be changed. Switch from str.split() to shlex.split() for more robust command-line argument parsing.
[feature]
Implement a .cwd (current working directory) suboption for post-write hooks (of type console_scripts). This is useful for tools like pre-commit, which rely on the working directory to locate the necessary config files. Add pre-commit as an example to the documentation. Minor change: rename some variables from ticket 819 to improve readability.
bug
[bug] [autogenerate]
Refactored the implementation of MigrateOperation constructs such as CreateIndexOp, CreateTableOp, etc. so that they no longer rely upon maintaining a persistent version of each schema object internally; instead, the state variables of each operation object will be used to produce the corresponding construct when the operation is invoked. The rationale is so that environments which make use of operation-manipulation schemes such as those those discussed in Fine-Grained Autogenerate Generation with Rewriters are better supported, allowing end-user code to manipulate the public attributes of these objects which will then be expressed in the final output, an example is some_create_index_op.kw["postgresql_concurrently"] = True.
Previously, these objects when generated from autogenerate would typically hold onto the original, reflected element internally without honoring the other state variables of each construct, preventing the public API from working.
[bug] [environment]
Fixed regression caused by the SQLAlchemy 1.4/2.0 compatibility switch where calling .rollback() or .commit() explicitly within the context.begin_transaction() context manager would cause it to fail when the block ended, as it did not expect that the transaction was manually closed.
[bug] [autogenerate]
Improved the rendering of op.add_column() operations when adding multiple columns to an existing table, so that the order of these statements matches the order in which the columns were declared in the application’s table metadata. Previously the added columns were being sorted alphabetically.
[bug] [versioning]
The algorithm used for calculating downgrades/upgrades/iterating revisions has been rewritten, to resolve ongoing issues of branches not being handled consistently particularly within downgrade operations, as well as for overall clarity and maintainability. This change includes that a deprecation warning is emitted if an ambiguous command such as “downgrade -1” when multiple heads are present is given.
In particular, the change implements a long-requested use case of allowing downgrades of a single branch to a branchpoint.
Huge thanks to Simon Bowly for their impressive efforts in successfully tackling this very difficult problem.
[bug] [batch] ¶
Added missing batch_op.create_table_comment(), batch_op.drop_table_comment() directives to batch ops.
DBD::MariaDB is the Perl5 Database Interface driver for MariaDB
and MySQL databases. In other words: DBD::MariaDB is an interface
between the Perl programming language and the MariaDB/MySQL
programming API that comes with the MariaDB/MySQL relational database
management system. Most functions provided by this programming API
are supported. Some rarely used functions are missing, mainly
because no-one ever requested them. :-)
=== 5.50.0 (2021-11-01)
* Make Migrator :allow_missing_migration_files also allow down migrations
where the current database version is greater than the last migration file
version (francisconeves97) (#1789)
* Fix Model#freeze in composition, serialization, and
serialization_modification_detection plugins to return self (jeremyevans)
(#1788)
* Fix typecasting of lazy columns when using lazy_attributes plugin in model
where dataset selects from subquery (jeremyevans)
* Add :before_preconnect Database option, for configuring extensions loaded
via :preconnect_extensions (MarcPer, jeremyevans) (#1786)
* Change Dataset#columns! to use a LIMIT 0 query instead of a LIMIT 1 query
(jeremyevans)
* Add sql_log_normalizer extension for normalizing logged SQL, helpful for
analytics and sensitive data (jeremyevans)
* Add support for range_merge, multirange, and unnest, and PGMultiRange#op
to pg_range_ops extension (jeremyevans)
* Add pg_multirange extension with support for PostgreSQL 14+ multirange
types (jeremyevans)
=== 5.49.0 (2021-10-01)
* Switch block_given? usage to defined?(yield) (jeremyevans)
* Support table aliases for JOIN USING columns on PostgreSQL 14+
(jeremyevans)
* Support calling PostgreSQL procedures without arguments (jeremyevans)
* Support hstore subscripts in pg_hstore_ops on PostgreSQL 14+, for updating
only part of an hstore value (jeremyevans)
* Support JSONB subscripts in pg_json_ops on PostgreSQL 14+, for updating
only part of a JSONB value (jeremyevans)
* Support SQL::Expression#sequel_ast_transform for custom AST transforms on
arbitrary expressions (jeremyevans)
* Add Database#create_trigger :replace option on PostgreSQL 14+ for CREATE
OR REPLACE TRIGGER (jeremyevans)
* Make auto_validations plugin automatically setup no_null_byte validations
(jeremyevans)
* Add Model#validates_no_null_byte to validation_helpers plugin
(jeremyevans)
A PostgreSQL SQL syntax beautifier that can work as a console program or as a
CGI.
This SQL formatter/beautifier supports keywords from SQL-92, SQL-99, SQL-2003,
SQL-2008, SQL-2011 and PostgreSQL specifics keywords. Works with any other
databases too.
Changes from the submitted version in the PR:
- p5 prefix for the package name
- clean up of github logic
- REPLACE_PERL
- updated distinfo with BLAKE2s
