SQLite Release 3.31.1:
Revert the data layout for an internal-use-only SQLite data structure. Applications that use SQLite should never reference internal SQLite data structures, but some do anyhow, and a change to one such data structure in 3.30.0 broke a popular and widely-deployed application. Reverting that change in SQLite, at least temporarily, gives developers of misbehaving applications time to fix their code.
Fix a typos in the sqlite3ext.h header file that prevented the sqlite3_stmt_isexplain() and sqlite3_value_frombind() interfaces from being called from run-time loadable extensions.
SQLite Release 3.31.0:
Add support for generated columns.
Add the sqlite3_hard_heap_limit64() interface and the corresponding PRAGMA hard_heap_limit command.
Enhance the function_list pragma to show the number of arguments on each function, the type of function (scalar, aggregate, window), and the function property flags SQLITE_DETERMINISTIC, SQLITE_DIRECTONLY, SQLITE_INNOCUOUS, and/or SQLITE_SUBTYPE.
Add the aggregated mode feature to the DBSTAT virtual table.
Add the SQLITE_OPEN_NOFOLLOW option to sqlite3_open_v2() that prevents SQLite from opening symbolic links.
Added the "#-N" array notation for JSON function path arguments.
Added the SQLITE_DBCONFIG_TRUSTED_SCHEMA connection setting which is also controllable via the new trusted_schema pragma and at compile-time using the -DSQLITE_TRUSTED_SCHEMA compile-time option.
Added APIs sqlite3_filename_database(), sqlite3_filename_journal(), and sqlite3_filename_wal() which are useful for specialized extensions.
Add the sqlite3_uri_key() interface.
Upgraded the sqlite3_uri_parameter() function so that it works with the rollback journal or WAL filename in addition to the database filename.
Provide the ability to tag application-defined SQL functions with new properties SQLITE_INNOCUOUS or SQLITE_DIRECTONLY.
Add new verbs to sqlite3_vtab_config() so that the xConnect method of virtual tables can declare the virtual table as SQLITE_VTAB_INNOCUOUS or SQLITE_VTAB_DIRECTONLY.
Faster response to sqlite3_interrupt().
Added the uuid.c extension module implementing functions for processing RFC-4122 UUIDs.
The lookaside memory allocator is enhanced to support two separate memory pools with different sized allocations in each pool. This allows more memory allocations to be covered by lookaside while at the same time reducing the heap memory usage to 48KB per connection, down from 120KB.
The legacy_file_format pragma is deactivated. It is now a no-op. In its place, the SQLITE_DBCONFIG_LEGACY_FILE_FORMAT option to sqlite3_db_config() is provided. The legacy_file_format pragma is deactivated because (1) it is rarely useful and (2) it is incompatible with VACUUM in schemas that have tables with both generated columns and descending indexes.
SQLite Release 3.30.1:
Fix a bug in the query flattener that might cause a segfault for nested queries that use the new FILTER clause on aggregate functions.
Cherrypick fixes for other obscure problems found since the 3.30.0 release.
SQLite Release 3.30.0
Add support for the FILTER clause on aggregate functions.
Add support for the NULLS FIRST and NULLS LAST syntax in ORDER BY clauses.
The index_info and index_xinfo pragmas are enhanced to provide information about the on-disk representation of WITHOUT ROWID tables.
Add the sqlite3_drop_modules() interface, allowing applications to disable automatically loaded virtual tables that they do not need.
Improvements to the .recover dot-command in the CLI so that it recovers more content from corrupt database files.
Enhance the RBU extension to support indexes on expressions.
Change the schema parser so that it will error out if any of the type, name, and tbl_name columns of the sqlite_master table have been corrupted and the database connection is not in writable_schema mode.
The PRAGMA function_list, PRAGMA module_list, and PRAGMA pragma_list commands are now enabled in all builds by default. Disable them using -DSQLITE_OMIT_INTROSPECTION_PRAGMAS.
Add the SQLITE_DBCONFIG_ENABLE_VIEW option for sqlite3_db_config().
Added the TCL Interface config method in order to be able to disable SQLITE_DBCONFIG_ENABLE_VIEW as well as control other sqlite3_db_config() options from TCL.
Added the SQLITE_DIRECTONLY flag for application-defined SQL functions to prevent those functions from being used inside triggers and views.
The legacy SQLITE_ENABLE_STAT3 compile-time option is now a no-op.
SQLite Release 3.29.0:
Added the SQLITE_DBCONFIG_DQS_DML and SQLITE_DBCONFIG_DQS_DDL actions to sqlite3_db_config() for activating and deactivating the double-quoted string literal misfeature. Both default to "on" for legacy compatibility, but developers are encouraged to turn them "off", perhaps using the -DSQLITE_DQS=0 compile-time option.
-DSQLITE_DQS=0 is now a recommended compile-time option.
Improvements to the query planner:
Improved optimization of AND and OR operators when one or the other operand is a constant.
Enhancements to the LIKE optimization for cases when the left-hand side column has numeric affinity.
Added the "sqlite_dbdata" virtual table for extracting raw low-level content from an SQLite database, even a database that is corrupt.
Enhancements to the CLI:
Add the ".recover" command which tries to recover as much content as possible from a corrupt database file.
Add the ".filectrl" command useful for testing.
Add the long-standing ".testctrl" command to the ".help" menu.
Added the ".dbconfig" command
SQLite Release 3.28.0:
Enhanced window functions:
Add support the EXCLUDE clause.
Add support for window chaining.
Add support for GROUPS frames.
Add support for "<expr> PRECEDING" and "<expr> FOLLOWING" boundaries in RANGE frames.
Added the new sqlite3_stmt_isexplain(S) interface for determining whether or not a prepared statement is an EXPLAIN.
Enhanced VACUUM INTO so that it works for read-only databases.
New query optimizations:
Enable the LIKE optimization for cases when the ESCAPE keyword is present and PRAGMA case_sensitive_like is on.
In queries that are driven by a partial index, avoid unnecessary tests of the constraint named in the WHERE clause of the partial index, since we know that constraint must always be true.
Enhancements to the TCL Interface:
Added the -returntype option to the function method.
Added the new bind_fallback method.
Enhancements to the CLI:
Added support for bound parameters and the .parameter command.
Fix the readfile() function so that it returns an empty BLOB rather than throwing an out-of-memory error when reading an empty file.
Fix the writefile() function so that when it creates new directories along the path of a new file, it gives them umask permissions rather than the same permissions as the file.
Change --update option in the .archive command so that it skips files that are already in the archive and are unchanged. Add the new --insert option that works like --update used to work.
Added the fossildelta.c extension that can create, apply, and deconstruct the Fossil DVCS file delta format that is used by the RBU extension.
Added the SQLITE_DBCONFIG_WRITABLE_SCHEMA verb for the sqlite3_db_config() interface, that does the same work as PRAGMA writable_schema without using the SQL parser.
Added the sqlite3_value_frombind() API for determining if the argument to an SQL function is from a bound parameter.
Security and compatibilities enhancements to fts3_tokenizer():
The fts3_tokenizer() function always returns NULL unless either the legacy application-defined FTS3 tokenizers interface are enabled using the sqlite3_db_config(SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER) setting, or unless the first argument to fts3_tokenizer() is a bound parameter.
The two-argument version of fts3_tokenizer() accepts a pointer to the tokenizer method object even without the sqlite3_db_config(SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER) setting if the second argument is a bound parameter
Improved robustness against corrupt database files.
Miscellaneous performance enhancements
SQLite Release 3.27.2:
Fix a bug in the IN operator that was introduced by an attempted optimization in version 3.27.0.
Fix a bug causing a crash when a window function is misused.
Fix various documentation typos
SQLite Release 3.27.1:
Fix a bug in the query optimizer: an adverse interaction between the OR optimization and the optimization that tries to use values read directly from an expression index instead of recomputing the expression.
Hashes:
Changes carried forward from version 3.27.0:
Added the VACUUM INTO command
Issue an SQLITE_WARNING message on the error log if a double-quoted string literal is used.
The sqlite3_normalized_sql() interface works on any prepared statement created using sqlite3_prepare_v2() or sqlite3_prepare_v3(). It is no longer necessary to use sqlite3_prepare_v3() with SQLITE_PREPARE_NORMALIZE in order to use sqlite3_normalized_sql().
Added the remove_diacritics=2 option to FTS3 and FTS5.
Added the SQLITE_PREPARE_NO_VTAB option to sqlite3_prepare_v3(). Use that option to prevent circular references to shadow tables from causing resource leaks.
Enhancements to the sqlite3_deserialize() interface:
Add the SQLITE_FCNTL_SIZE_LIMIT file-control for setting an upper bound on the size of the in-memory database created by sqlite3_deserialize. The default upper bound is 1GiB, or whatever alternative value is specified by sqlite3_config(SQLITE_CONFIG_MEMDB_MAXSIZE) and/or SQLITE_MEMDB_DEFAULT_MAXSIZE.
Honor the SQLITE_DESERIALIZE_READONLY flag, which was previously described in the documentation, but was previously a no-op.
Enhance the "deserialize" command of the TCL Interface to give it new "--maxsize N" and "--readonly BOOLEAN" options.
Enhancements to the CLI, mostly to support testing and debugging of the SQLite library itself:
Add support for ".open --hexdb". The "dbtotxt" utility program used to generate the text for the "hexdb" is added to the source tree.
Add support for the "--maxsize N" option on ".open --deserialize".
Add the "--memtrace" command-line option, to show all memory allocations and deallocations.
Add the ".eqp trace" option on builds with SQLITE_DEBUG, to enable bytecode program listing with indentation and PRAGMA vdbe_trace all in one step.
Add the ".progress" command for accessing the sqlite3_progress_handler() interface.
Add the "--async" option to the ".backup" command.
Add options "--expanded", "--normalized", "--plain", "--profile", "--row", "--stmt", and "--close" to the ".trace" command.
Increased robustness against malicious SQL that is run against a maliciously corrupted database.
Bug fixes:
Do not use a partial index to do a table scan on an IN operator.
Fix the query flattener so that it works on queries that contain subqueries that use window functions.
Ensure that ALTER TABLE modifies table and column names embedded in WITH clauses that are part of views and triggers.
Fix a parser bug that prevented the use of parentheses around table-valued functions.
Fix a problem with the OR optimization on indexes on expressions.
Fix a problem with the LEFT JOIN strength reduction optimization in which the optimization was being applied inappropriately due to an IS NOT NULL operator.
Fix the REPLACE command so that it is no longer able to sneak a NULL value into a NOT NULL column even if the NOT NULL column has a default value of NULL.
Fix a problem with the use of window functions used within correlated subqueries.
Fix the ALTER TABLE RENAME COLUMN command so that it works for tables that have redundant UNIQUE constraints.
Fix a bug that caused zeroblob values to be truncated when inserted into a table that uses an expression index.
SQLite Release 3.26.0:
Optimization: When doing an UPDATE on a table with indexes on expressions, do not update the expression indexes if they do not refer to any of the columns of the table being updated.
Allow the xBestIndex() method of virtual table implementations to return SQLITE_CONSTRAINT to indicate that the proposed query plan is unusable and should not be given further consideration.
Added the SQLITE_DBCONFIG_DEFENSIVE option which disables the ability to create corrupt database files using ordinary SQL.
Added support for read-only shadow tables when the SQLITE_DBCONFIG_DEFENSIVE option is enabled.
Added the PRAGMA legacy_alter_table command, which if enabled causes the ALTER TABLE command to behave like older version of SQLite (prior to version 3.25.0) for compatibility.
Added PRAGMA table_xinfo that works just like PRAGMA table_info except that it also shows hidden columns in virtual tables.
Added the explain virtual table as a run-time loadable extension.
Add a limit counter to the query planner to prevent excessive sqlite3_prepare() times for certain pathological SQL inputs.
Added support for the sqlite3_normalized_sql() interface, when compiling with SQLITE_ENABLE_NORMALIZE.
Enhanced triggers so that they can use table-valued functions that exist in schemas other than the schema where the trigger is defined.
Enhancements to the CLI:
- Improvements to the ".help" command.
- The SQLITE_HISTORY environment variable, if it exists, specifies the name of the command-line editing history file
- The --deserialize option associated with opening a new database cause the database file to be read into memory and accessed using the sqlite3_deserialize() API. This simplifies running tests on a database without modifying the file on disk.
Enhancements to the geopoly extension:
- Aways stores polygons using the binary format, which is faster and uses less space.
- Added the geopoly_regular() function.
- Added the geopoly_ccw() function.
Enhancements to the session extension:
- Added the SQLITE_CHANGESETAPPLY_INVERT flag
- Added the sqlite3changeset_start_v2() interface and the SQLITE_CHANGESETSTART_INVERT flag.
- Added the changesetfuzz.c test-case generator utility.
SQLite Release 3.25.3:
Disallow the use of window functions in the recursive part of a CTE.
Fix the behavior of typeof() and length() on virtual tables.
Strengthen defenses against deliberately corrupted database files.
Fix a problem in the query planner that results when a row-value expression is used with a PRIMARY KEY with redundant columns.
Fix the query planner so that it works correctly for IS NOT NULL operators in the ON clause of a LEFT JOIN with the SQLITE_ENABLE_STAT4 compile-time option.
SQLite Release 3.25.2:
Add the PRAGMA legacy_alter_table=ON command that causes the "ALTER TABLE RENAME" command to behave as it did in SQLite versions 3.24.0 and earlier: references to the renamed table inside the bodies of triggers and views are not updated. This new pragma provides a compatibility work around for older programs that expected the older, wonky behavior of ALTER TABLE RENAME.
Fix a problem with the new window functions implementation that caused a malfunction when complicated expressions involving window functions were used inside of a view.
Fixes for various other compiler warnings and minor problems associated with obscure configurations.
SQLite Release 3.25.1:
Extra sanity checking added to ALTER TABLE in the 3.25.0 release sometimes raises a false-positive when the table being modified has a trigger that updates a virtual table. The false-positive caused the ALTER TABLE to rollback, thus leaving the schema unchanged.
The fix in the 3.25.0 release for the endless-loop in the byte-code associated with the ORDER BY LIMIT optimization did not work for some queries involving window functions. An additional correction is required.
SQLite Release 3.25.0:
Add support for window functions
Enhancements the ALTER TABLE command:
- Add support for renaming columns within a table using ALTER TABLE table RENAME COLUMN oldname TO newname.
- Fix table rename feature so that it also updates references to the renamed table in triggers and views.
Query optimizer improvements:
- Avoid unnecessary loads of columns in an aggregate query that are not within an aggregate function and that are not part of the GROUP BY clause.
- The IN-early-out optimization: When doing a look-up on a multi-column index and an IN operator is used on a column other than the left-most column, then if no rows match against the first IN value, check to make sure there exist rows that match the columns to the right before continuing with the next IN value.
- Use the transitive property to try to propagate constant values within the WHERE clause. For example, convert "a=99 AND b=a" into "a=99 AND b=99".
Use a separate mutex on every inode in the unix VFS, rather than a single mutex shared among them all, for slightly better concurrency in multi-threaded environments.
Enhance the PRAGMA integrity_check command for improved detection of problems on the page freelist.
Output infinity as 1e999 in the ".dump" command of the command-line shell.
Added the SQLITE_FCNTL_DATA_VERSION file-control.
Added the Geopoly module
Bug fixes:
The ORDER BY LIMIT optimization might have caused an infinite loop in the byte code of the prepared statement under very obscure circumstances, due to a confluence of minor defects in the query optimizer.
On an UPSERT when the order of constraint checks is rearranged, ensure that the affinity transformations on the inserted content occur before any of the constraint checks.
Avoid using a prepared statement for ".stats on" command of the CLI after it has been closed by the ".eqp full" logicc.
The LIKE optimization was generating incorrect byte-code and hence getting the wrong answer if the left-hand operand has numeric affinity and the right-hand-side pattern is '/%' or if the pattern begins with the ESCAPE character.
SQLite Release 3.24.0:
Add support for PostgreSQL-style UPSERT.
Add support for auxiliary columns in r-tree tables.
Add C-language APIs for discovering SQL keywords used by SQLite: sqlite3_keyword_count(), sqlite3_keyword_name(), and sqlite3_keyword_check().
Add C-language APIs for dynamic strings based on the sqlite3_str object.
Enhance ALTER TABLE so that it recognizes "true" and "false" as valid arguments to DEFAULT.
Add the sorter-reference optimization as a compile-time option. Only available if compiled with SQLITE_ENABLE_SORTER_REFERENCES.
Improve the format of the EXPLAIN QUERY PLAN raw output, so that it gives better information about the query plan and about the relationships between the various components of the plan.
Added the SQLITE_DBCONFIG_RESET_DATABASE option to the sqlite3_db_config() API.
CLI Enhancements:
Automatically intercepts the raw EXPLAIN QUERY PLAN output and reformats it into an ASCII-art graph.
Lines that begin with "#" and that are not in the middle of an SQL statement are interpreted as comments.
Added the --append option to the ".backup" command.
Added the ".dbconfig" command.
Performance:
UPDATE avoids unnecessary low-level disk writes when the contents of the database file do not actually change. For example, "UPDATE t1 SET x=25 WHERE y=?" generates no extra disk I/O if the value in column x is already 25. Similarly, when doing UPDATE on records that span multiple pages, only the subset of pages that actually change are written to disk. This is a low-level performance optimization only and does not affect the behavior of TRIGGERs or other higher level SQL structures.
Queries that use ORDER BY and LIMIT now try to avoid computing rows that cannot possibly come in under the LIMIT. This can greatly improve performance of ORDER BY LIMIT queries, especially when the LIMIT is small relative to the number of unrestricted output rows.
The OR optimization is allowed to proceed even if the OR expression has also been converted into an IN expression. Uses of the OR optimization are now also more clearly shown in the EXPLAIN QUERY PLAN output.
The query planner is more aggressive about using automatic indexes for views and subqueries for which it is not possible to create a persistent index.
Make use of the one-pass UPDATE and DELETE query plans in the R-Tree extension where appropriate.
Performance improvements in the LEMON-generated parser.
Bug fixes:
For the right-hand table of a LEFT JOIN, compute the values of expressions directly rather than loading precomputed values out of an expression index as the expression index might not contain the correct value.
Do not attempt to use terms from the WHERE clause to enable indexed lookup of the right-hand table of a LEFT JOIN.
Fix a memory leak that can occur following a failure to open error in the CSV virtual table
Fix a long-standing problem wherein a corrupt schema on the sqlite_sequence table used by AUTOINCREMENT can lead to a crash.
Fix the json_each() function so that it returns valid results on its "fullkey" column when the input is a simple value rather than an array or object.
SQLite Release 3.23.1:
Fix two problems in the new LEFT JOIN strength reduction optimization.
Fix misbehavior of the FTS5 xBestIndex method.
Fix a harmless reference to an uninitialized virtual machine register.
Fix the CLI so that it builds with -DSQLITE_UNTESTABLE
Fix the eval.c extension so that it works with PRAGMA empty_result_callbacks=ON.
Fix the generate_series virtual table so that it correctly returns no rows if any of its constraints are NULL.
Performance enhancements in the parser.
SQLite Release 3.23.0:
Add the sqlite3_serialize() and sqlite3_deserialize() interfaces when the SQLITE_ENABLE_DESERIALIZE compile-time option is used.
Recognize TRUE and FALSE as constants. (For compatibility, if there exist columns named "true" or "false", then the identifiers refer to the columns rather than Boolean constants.)
Support operators IS TRUE, IS FALSE, IS NOT TRUE, and IS NOT FALSE.
Added the SQLITE_DBSTATUS_CACHE_SPILL option to sqlite3_db_status() for reporting the number of cache spills that have occurred.
The "alternate-form-2" flag ("!") on the built-in printf implementation now causes string substitutions to measure the width and precision in characters instead of bytes.
If the xColumn method in a virtual table implementation returns an error message using sqlite3_result_error() then give that error message preference over internally-generated messages.
Added the -A command-line option to the CLI to make it easier to manage SQLite Archive files.
Add support for INSERT OR REPLACE, INSERT OR IGNORE, and UPDATE OR REPLACE in the Zipfile virtual table.
Enhance the sqlite3changeset_apply() interface so that it is hardened against attacks from deliberately corrupted changeset objects.
Added the sqlite3_normalize() extension function.
Query optimizer enhancements:
Improve the omit-left-join optimization so that it works in cases where the right-hand table is UNIQUE but not necessarily NOT NULL.
Improve the push-down optimization so that it works for many LEFT JOINs.
Add the LEFT JOIN strength reduction optimization that converts a LEFT JOIN into an ordinary JOIN if there exist terms in the WHERE clause that would prevent the extra all-NULL row of the LEFT JOIN from appearing in the output set.
Avoid unnecessary writes to the sqlite_sequence table when an AUTOINCREMENT table is updated with an rowid that is less than the maximum.
Bug fixes:
Fix the parser to accept valid row value syntax.
Fix the query planner so that it takes into account dependencies in the arguments to table-valued functions in subexpressions in the WHERE clause.
Fix incorrect result with complex OR-connected WHERE and STAT4.
Fix potential corruption in indexes on expressions due to automatic datatype conversions.
Assertion fault in FTS4.
Incorrect result on the less-than operator in row values.
Always interpret non-zero floating-point values as TRUE, even if the integer part is zero.
Fix an issue in the fsdir(PATH) table-valued function to the fileio.c extension, that caused a segfault if the fsdir() table was used as the inner table of a join.
Issue an error rather instead of an assertion-fault or null-pointer dereference when the sqlite_master table is corrupted so that the sqlite_sequence table root page is really a btree-index page.
Fix the ANALYZE command so that it computes statistics on tables whose names begin with "sqlite".
Additional fixes for issues detected by OSSFuzz:
Fix a possible infinite loop on VACUUM for corrupt database files.
Disallow parameters in the WITH clause of triggers and views.
Fix a potential memory leak in row value processing.
Improve the performance of the replace() SQL function for cases where there are many substitutions on megabyte-sized strings, in an attempt to avoid OSSFuzz timeouts during testing.
Provide an appropriate error message when the sqlite_master table contains a CREATE TABLE AS statement. Formerly this caused either an assertion fault or null pointer dereference. Problem found by OSSFuzz on the GDAL project.
Incorrect assert() statement removed.
Fix a problem with using the LIKE optimization on an INTEGER PRIMARY KEY.
Release 3.22.0:
The output of sqlite3_trace_v2() now shows each individual SQL statement run within a trigger.
Add the ability to read from WAL mode databases even if the application lacks write permission on the database and its containing directory, as long as the -shm and -wal files exist in that directory.
Added the rtreecheck() scalar SQL function to the R-Tree extension.
Added the sqlite3_vtab_nochange() and sqlite3_value_nochange() interfaces to help virtual table implementations optimize UPDATE operations.
Added the sqlite3_vtab_collation() interface.
Added support for the "^" initial token syntax in FTS5.
New extensions:
The Zipfile virtual table can read and write a ZIP Archive.
Added the fsdir(PATH) table-valued function to the fileio.c extension, for listing the files in a directory.
The sqlite_btreeinfo eponymous virtual table for introspecting and estimating the sizes of the btrees in a database.
The Append VFS is a VFS shim that allows an SQLite database to be appended to some other file. This allows (for example) a database to be appended to an executable that then opens and reads the database.
Query planner enhancements:
The optimization that uses an index to quickly compute an aggregate min() or max() is extended to work with indexes on expressions.
The decision of whether to implement a FROM-clause subquery as a co-routine or using query flattening now considers whether the result set of the outer query is "complex" (if it contains functions or expression subqueries). A complex result set biases the decision toward the use of co-routines.
The planner avoids query plans that use indexes with unknown collating functions.
The planner omits unused LEFT JOINs even if they are not the right-most joins of a query.
Other performance optimizations:
A smaller and faster implementation of text to floating-point conversion subroutine: sqlite3AtoF().
The Lemon parser generator creates a faster parser.
Use the strcspn() C-library routine to speed up the LIKE and GLOB operators.
Improvements to the command-line shell:
The ".schema" command shows the structure of virtual tables.
Added support for reading and writing SQL Archive files using the .archive command.
Added the experimental .expert command
Added the ".eqp trigger" variant of the ".eqp" command
Enhance the ".lint fkey-indexes" command so that it works with WITHOUT ROWID tables.
If the filename argument to the shell is a ZIP archive rather than an SQLite database, then the shell automatically opens that ZIP archive using the Zipfile virtual table.
Added the edit() SQL function.
Added the .excel command to simplify exporting database content to a spreadsheet.
Databases are opened using Append VFS when the --append flag is used on the command line or with the .open command.
Enhance the SQLITE_ENABLE_UPDATE_DELETE_LIMIT compile-time option so that it works for WITHOUT ROWID tables.
Provide the sqlite_offset(X) SQL function that returns the byte offset into the database file to the beginning of the record holding value X, when compiling with -DSQLITE_ENABLE_OFFSET_SQL_FUNC.
Bug fixes
Release 3.21.0:
Take advantage of the atomic-write capabilities in the F2FS filesystem when available, for greatly reduced transaction overhead. This currently requires the SQLITE_ENABLE_BATCH_ATOMIC_WRITE compile-time option.
Allow ATTACH and DETACH commands to work inside of a transaction.
Allow WITHOUT ROWID virtual tables to be writable if the PRIMARY KEY contains exactly one column.
The "fsync()" that occurs after the header is written in a WAL reset now uses the sync settings for checkpoints. This means it will use a "fullfsync" on macs if PRAGMA checkpoint_fullfsync set on.
The sqlite3_sourceid() function tries to detect if the source code has been modified from what is checked into version control and if there are modifications, the last four characters of the version hash are shown as "alt1" or "alt2". The objective is to detect accidental and/or careless edits. A forger can subvert this feature.
Improved de-quoting of column names for CREATE TABLE AS statements with an aggregate query on the right-hand side.
Fewer "stat()" system calls issued by the unix VFS.
Enhanced the LIKE optimization so that it works with an ESCAPE clause.
Enhanced PRAGMA integrity_check and PRAGMA quick_check to detect obscure row corruption that they were formerly missing. Also update both pragmas so that they return error text rather than SQLITE_CORRUPT when encountering corruption in records.
The query planner now prefers to implement FROM-clause subqueries using co-routines rather using the query flattener optimization. Support for the use of co-routines for subqueries may no longer be disabled.
Pass information about !=, IS, IS NOT, NOT NULL, and IS NULL constraints into the xBestIndex method of virtual tables.
Enhanced the CSV virtual table so that it accepts the last row of input if the final new-line character is missing.
Remove the rarely-used "scratch" memory allocator. Replace it with the SQLITE_CONFIG_SMALL_MALLOC configuration setting that gives SQLite a hint that large memory allocations should be avoided when possible.
Added the swarm virtual table to the existing union virtual table extension.
Added the sqlite_dbpage virtual table for providing direct access to pages of the database file. The source code is built into the amalgamation and is activated using the -DSQLITE_ENABLE_DBPAGE_VTAB compile-time option.
Add a new type of fts5vocab virtual table - "instance" - that provides direct access to an FTS5 full-text index at the lowest possible level.
Remove a call to rand_s() in the Windows VFS since it was causing problems in Firefox on some older laptops.
The src/shell.c source code to the command-line shell is no longer under version control. That file is now generated as part of the build process.
Miscellaneous microoptimizations reduce CPU usage by about 2.1%.
Bug fixes
1. Update the text of error messages returned by sqlite3_errmsg() for some error codes.
2. Add new pointer passing interfaces.
3. Backwards-incompatible changes to some extensions in order to take advantage of the improved security offered by the new pointer passing interfaces:
a. Extending FTS5 → requires sqlite3_bind_pointer() to find the fts5_api pointer.
b. carray(PTR,N) → requires sqlite3_bind_pointer() to set the PTR parameter.
c. remember(V,PTR) → requires sqlite3_bind_pointer() to set the PTR parameter.
4. Added the SQLITE_STMT virtual table extension.
5. Added the COMPLETION extension - designed to suggest tab-completions for interactive user interfaces. This is a work in progress. Expect further enhancements in future releases.
6. Added the UNION virtual table extension.
7. The built-in date and time functions have been enhanced so that they can be used in CHECK constraints, in indexes on expressions, and in the WHERE clauses of partial indexes, provided that they do not use the 'now', 'localtime', or 'utc' keywords. Futher information.
8. Added the sqlite3_prepare_v3() and sqlite3_prepare16_v3() interfaces with the extra "prepFlags" parameters.
9. Provide the SQLITE_PREPARE_PERSISTENT flag for sqlite3_prepare_v3() and use it to limit lookaside memory misuse by FTS3, FTS5, and the R-Tree extension.
10. Added the PRAGMA secure_delete=FAST command. When secure_delete is set to FAST, old content is overwritten with zeros as long as that does not increase the amount of I/O. Deleted content might still persist on the free-page list but will be purged from all b-tree pages.
11. Enhancements to the command-line shell:
a. Add support for tab-completion using the COMPLETION extension, for both readline and linenoise.
b. Add the ".cd" command.
c. Enhance the ".schema" command to show the schema of all attached databases.
d. Enhance ".tables" so that it shows the schema names for all attached if the name is anything other than "main".
e. The ".import" command ignores an initial UTF-8 BOM.
f. Added the "--newlines" option to the ".dump" command to cause U+000a and U+000d characters to be output literally rather than escaped using the replace() function.
12. Query planner enhancements:
a. When generating individual loops for each ORed term of an OR scan, move any constant WHERE expressions outside of the loop, as is done for top-level loops.
b. The query planner examines the values of bound parameters to help determine if a partial index is usable.
c. When deciding between two plans with the same estimated cost, bias the selection toward the one that does not use the sorter.
d. Evaluate WHERE clause constraints involving correlated subqueries last, in the hope that they never have be evaluated at all.
e. Do not use the flattening optimization for a sub-query on the RHS of a LEFT JOIN if that subquery reads data from a virtual table as doing so prevents the query planner from creating automatic indexes on the results of the sub-query, which can slow down the query.
13. Add SQLITE_STMTSTATUS_REPREPARE, SQLITE_STMTSTATUS_RUN, and SQLITE_STMTSTATUS_MEMUSED options for the sqlite3_stmt_status() interface.
14. Provide PRAGMA functions for PRAGMA integrity_check, PRAGMA quick_check, and PRAGMA foreign_key_check.
15. Add the -withoutnulls option to the TCL interface eval method.
16. Enhance the sqlite3_analyzer.exe utility program so that it shows the number of bytes of metadata on btree pages.
17. The SQLITE_DBCONFIG_ENABLE_QPSG run-time option and the SQLITE_ENABLE_QPSG compile-time option enable the query planner stability guarantee. See also ticket 892fc34f173e99d8
18. Miscellaneous optimizations result in a 2% reduction in CPU cycles used.
Bug Fixes:
19. Fix the behavior of sqlite3_column_name() for queries that use the flattening optimization so that the result is consistent with other queries that do not use that optimization, and with PostgreSQL, MySQL, and SQLServer. Ticket de3403bf5ae.
20. Fix the query planner so that it knows not to use automatic indexes on the right table of LEFT JOIN if the WHERE clause uses the IS operator. Fix for ce68383bf6aba.
21. Ensure that the query planner knows that any column of a flattened LEFT JOIN can be NULL even if that column is labeled with "NOT NULL". Fix for ticket 892fc34f173e99d8.
22. Fix rare false-positives in PRAGMA integrity_check when run on a database connection with attached databases. Ticket a4e06e75a9ab61a12
23. Fix a bug (discovered by OSSFuzz) that causes an assertion fault if certain dodgy CREATE TABLE declarations are used.
Fix more bugs in the LEFT JOIN flattening optimization. Ticket 7fde638e94287d2c.
Changes carried forward from version 3.19.1 (2017-05-24):
Fix a bug in the LEFT JOIN flattening optimization. Ticket cad1ab4cb7b0fc.
Remove a surplus semicolon that was causing problems for older versions of MSVC.
---------------------
The SQLITE_READ authorizer callback is invoked once with a column name that is an empty string for every table referenced in a query from which no columns are extracted.
When using an index on an expression, try to use expression values already available in the index, rather than loading the original columns and recomputing the expression.
Enhance the flattening optimization so that it is able to flatten views on the right-hand side of a LEFT JOIN.
Use replace() instead of char() for escaping newline and carriage-return characters embedded in strings in the .dump output from the command-line shell.
Avoid unnecessary foreign key processing in UPDATE statements that do not touch the columns that are constrained by the foreign keys.
On a DISTINCT query that uses an index, try to skip ahead to the next distinct entry using the index rather than stepping through rows, when an appropriate index is available.
Avoid unnecessary invalidation of sqlite3_blob handles when making changes to unrelated tables.
Transfer any terms of the HAVING clause that use only columns mentioned in the GROUP BY clause over to the WHERE clause for faster processing.
Reuse the same materialization of a VIEW if that VIEW appears more than once in the same query.
Enhance PRAGMA integrity_check so that it identifies tables that have two or more rows with the same rowid.
Enhance the FTS5 query syntax so that column filters may be applied to arbitrary expressions.
Enhance the json_extract() function to cache and reuse parses of JSON input text.
Added the anycollseq.c loadable extension that allows a generic SQLite database connection to read a schema that contains unknown and/or application-specific collating sequences.
Bug Fixes:
----------
Fix a problem in REPLACE that can result in a corrupt database containing two or more rows with the same rowid.
Fix a problem in PRAGMA integrity_check that was causing a subsequent VACUUM to behave suboptimally.
Fix the PRAGMA foreign_key_check command so that it works correctly with foreign keys on WITHOUT ROWID tables.
Disallow leading zeros in numeric constants in JSON.
Disallow control characters inside of strings in JSON.
Limit the depth of recursion for JSON objects and arrays in order to avoid excess stack usage in the recursive descent parser.
Added the PRAGMA optimize command
The SQLite version identifier returned by the sqlite_source_id() SQL function and the sqlite3_sourceid() C API and found in the SQLITE_SOURCE_ID macro is now a 64-digit SHA3-256 hash instead of a 40-digit SHA1 hash.
Added the json_patch() SQL function to the JSON1 extension.
Enhance the LIKE optimization so that it works for arbitrary expressions on the left-hand side as long as the LIKE pattern on the right-hand side does not begin with a digit or minus sign.
Added the sqlite3_set_last_insert_rowid() interface and use the new interface in the FTS3, FTS4, and FTS5 extensions to ensure that the sqlite3_last_insert_rowid() interface always returns reasonable values.
Enhance PRAGMA integrity_check and PRAGMA quick_check so that they verify CHECK constraints.
Enhance the query plans for joins to detect empty tables early and halt without doing unnecessary work.
Enhance the sqlite3_mprintf() family of interfaces and the printf SQL function to put comma separators at the thousands marks for integers, if the "," format modifier is used in between the "%" and the "d" (example: "%,d").
Added the -DSQLITE_MAX_MEMORY=N compile-time option.
Added the .sha3sum dot-command and the .selftest dot-command to the command-line shell
Begin enforcing SQLITE_LIMIT_VDBE_OP. This can be used, for example, to prevent excessively large prepared statements in systems that accept SQL queries from untrusted users.
Various performance improvements.
Bug Fixes:
----------
Ensure that indexed expressions with collating sequences are handled correctly. Fix for ticket eb703ba7b50c1a5.
Fix a bug in the 'start of ...' modifiers for the date and time functions. Ticket 6097cb92745327a1
Fix a potential segfault in complex recursive triggers, resulting from a bug in the OP_Once opcode introduced as part of a performance optimization in version 3.15.0. Ticket 06796225f59c057c
In the RBU extension, add extra sync operations to avoid the possibility of corruption following a power failure.
The sqlite3_trace_v2() output for nested SQL statements should always begin with a "--" comment marker.
* Approximately 25% better performance from the R-Tree extension.
* Uses compiler built-ins (ex: __builtin_bswap32() or _byteswap_ulong()) for byteswapping when available.
* Uses the sqlite3_blob key/value access object instead of SQL for pulling content out of R-Tree nodes
* Other miscellaneous enhancements such as loop unrolling.
* Add the SQLITE_DEFAULT_LOOKASIDE compile-time option.
* Increase the default lookaside size from 512,125 to 1200,100 as this provides better performance while only adding 56KB of extra memory per connection. Memory-sensitive applications can restore the old default at compile-time, start-time, or run-time.
* Use compiler built-ins __builtin_sub_overflow(), __builtin_add_overflow(), and __builtin_mul_overflow() when available. (All compiler built-ins can be omitted with the SQLITE_DISABLE_INTRINSIC compile-time option.)
* Added the SQLITE_ENABLE_NULL_TRIM compile-time option, which can result in significantly smaller database files for some applications, at the risk of being incompatible with older versions of SQLite.
* Change SQLITE_DEFAULT_PCACHE_INITSZ from 100 to 20, for improved performance.
* Added the SQLITE_UINT64_TYPE compile-time option as an analog to SQLITE_INT64_TYPE.
* Perform some UPDATE operations in a single pass instead of in two passes.
* Enhance the session extension to support WITHOUT ROWID tables.
* Fixed performance problems and potential stack overflows when creating views from multi-row VALUES clauses with hundreds of thousands of rows.
* Added the sha1.c extension.
* In the command-line shell, enhance the ".mode" command so that it restores the default column and row separators for modes "line", "list", "column", and "tcl".
* Enhance the SQLITE_DIRECT_OVERFLOW_READ option so that it works in WAL mode as long as the pages being read are not in the WAL file.
* Enhance the LEMON parser generator so that it can store the parser object as a stack variable rather than allocating space from the heap and make use of that enhancement in the amalgamation.
* Other performance improvements. Uses about 6.5% fewer CPU cycles.
Bug Fixes:
* Throw an error if the ON clause of a LEFT JOIN references tables to the right of the ON clause. This is the same behavior as PostgreSQL. Formerly, SQLite silently converted the LEFT JOIN into an INNER JOIN.
* Use the correct affinity for columns of automatic indexes.
* Ensure that the sqlite3_blob_reopen() interface can correctly handle short rows.
MASTER_SITES= site1 \
site2
style continuation lines to be simple repeated
MASTER_SITES+= site1
MASTER_SITES+= site2
lines. As previewed on tech-pkg. With thanks to rillig for fixing pkglint
accordingly.
Fix the REPLACE statement for WITHOUT ROWID tables that lack secondary indexes so that it works correctly with triggers and foreign keys. This was a new bug caused by performance optimizations added in version 3.16.0. Ticket 30027b613b4
Fix the sqlite3_value_text() interface so that it correctly translates content generated by zeroblob() into a string of all 0x00 characters. This is a long-standing issue discovered after the 3.16.1 release by OSS-Fuzz
Fix the bytecode generator to deal with a subquery in the FROM clause that is itself a UNION ALL where one side of the UNION ALL is a view that contains an ORDER BY. This is a long-standing issue that was discovered after the release of 3.16.1. See ticket 190c2507.
Adjust the sqlite3_column_count() API so it more often returns the same values for PRAGMA statements as it did in prior releases, to minimize disruption to applications that might be using that interface in unexpected ways.
Fix a bug concerning the use of row values within triggers (see ticket 8c9458e7) that was in version 3.15.0 but was not reported until moments after the 3.16.0 release was published.
Uses 9% fewer CPU cycles. (See the CPU performance measurement report for details on how the this performance increase was computed.)
Added experimental support for PRAGMA functions.
Added the SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE option to sqlite3_db_config().
Enhance the date and time functions so that the 'unixepoch' modifier works for the full span of supported dates.
Changed the default configuration of the lookaside memory allocator from 500 slots of 128 bytes each into 125 slots of 512 bytes each.
Enhanced "WHERE x NOT NULL" partial indexes so that they are usable if the "x" column appears in a LIKE or GLOB operator.
Enhanced sqlite3_interrupt() so that it interrupts checkpoint operations that are in process.
Enhanced the LIKE and GLOB matching algorithm to be faster for cases when the pattern contains multiple wildcards.
Added the SQLITE_FCNTL_WIN32_GET_HANDLE file control opcode.
Added ".mode quote" to the command-line shell.
Added ".lint fkey-indexes" to the command-line shell.
Added the .imposter dot-command to the command-line shell.
Added the remember(V,PTR) SQL function as a loadable extension.
Rename the SQLITE_OMIT_BUILTIN_TEST compile-time option to SQLITE_UNTESTABLE to better reflect the implications of using it.
Bug Fixes:
Fix a long-standing bug in the query planner that caused incorrect results on a LEFT JOIN where the left-hand table is a subquery and the join constraint is a bare column name coming from the left-hand subquery. Ticket 2df0107b.
Correctly handle the integer literal -0x8000000000000000 in the query planner.
This way they got added as dependencies to sqlite3.so, while
only the sqlite3 binary really needs libreadline.
However, even without the LDFLAGS, the sqlite3 binary correctly
links the library in for me (on NetBSD).
Addresses PR 50776.
Bump PKGREVISION.
* Multiple bug fixes to the row value logic that was introduced in version 3.15.0.
* Fix a NULL pointer dereference in ATTACH/DETACH following a maliciously constructed syntax error.
* Fix a crash that can occur following an out-of-memory condition in the built-in instr() function.
* In the JSON extension, fix the JSON validator so that it correctly rejects invalid backslash escapes within strings.
* Added SQLITE_FCNTL_WIN32_GET_HANDLE file control opcode.
Bug Fixes:
* Fix the VACUUM command so that it spills excess content to disk rather than holding everything in memory, and possible causing an out-of-memory error for larger database files. This fixes an issue introduced by version 3.15.0.
* Fix a case (present since 3.8.0 - 2013-08-26) where OR-connected terms in the ON clause of a LEFT JOIN might cause incorrect results.
* Fix a case where the use of row values in the ON clause of a LEFT JOIN might cause incorrect results.
pkgsrc changes
==============
Bind this package with databases/sqlite3 to get immediate updates along
with the sqlite3 package.
Set new versioning policy and concatenate versions of lemon and sqlite3.
The reasoning for it is as follows:
- currently lemon is an integral part of sqlite3
- lemon has its own conservative versioning regardless of changes in its code
- leave room for possible standalone lemon package with bumped versions
- reflect reality and make it human readable without checking the sources
Set LICENSE to public-domain.
Install documentation in the HTML format.
Compile with LDFLAGS set.
Finally don't use databases/sqlite3/Makefile.common as lemon is not
distributed in the same archive. Make use of Makefile.version that does the
same job of tracking upstream.
Do not set PKGCONFIG_OVERRIDE in Makefile.common, it is not accessible in
sqlite3-tcl.
Upstream changelog
==================
Changes are unknown, but at least the program acquired a -T command line
option used by brlcad.
Added support for row values.
Allow deterministic SQL functions in the WHERE clause of a partial index.
Added the "modeof=filename" URI parameter on the unix VFS
Added support for SQLITE_DBCONFIG_MAINDBNAME.
Added the ability to VACUUM an ATTACH-ed database.
Enhancements to the command-line shell:
Add the ".testcase" and ".check" dot-commands.
Added the --new option to the ".open" dot-command, causing any prior content in the database to be purged prior to opening.
Enhance the fts5vocab virtual table to handle "ORDER BY term" efficiently.
Miscellaneous micro-optimizations reduce CPU usage by more than 7% on common workloads. Most optimization in this release has been on the front-end (sqlite3_prepare_v2()).
Bug Fixes:
The multiply operator now correctly detects 64-bit integer overflow and promotes to floating point in all corner-cases.
Correct handling of columns with redundant unique indexes when those columns are used on the LHS of an IN operator.
Skip NULL entries on range queries in indexes on expressions.
Ensure that the AUTOINCREMENT counters in the sqlite_sequence table are initialized doing "Xfer Optimization" on "INSERT ... SELECT" statements.
Make sure the ORDER BY LIMIT optimization (from check-in 559733b09e) works with IN operators on INTEGER PRIMARY KEYs.
* Improved support for using the STDCALL calling convention in winsqlite3.dll.
* Fix the sqlite3_trace_v2() interface so that it is disabled if either the callback or the mask arguments are zero, in accordance with the documentation.
* Fix commenting errors and improve the comments generated on EXPLAIN listings when the -DSQLITE_ENABLE_EXPLAIN_COMMENTS compile-time option is used.
* Fix the ".read" command in the command-line shell so that it understands that its input is not interactive.
* Correct affinity computations for a SELECT on the RHS of an IN operator.
* The ORDER BY LIMIT optimization is not valid unless the inner-most IN operator loop is actually used by the query plan. Fix for ticket 0c4df46116e90f92.
* Fix an internal code generator problem that was causing some DELETE operations to no-op.
* A performance enhancement to the page-cache "truncate" operation reduces COMMIT time by dozens of milliseconds on systems with a large page cache.
* Fix to the --rbu option of sqldiff.
