SQLite Release 3.24.0:
Add support for PostgreSQL-style UPSERT.
Add support for auxiliary columns in r-tree tables.
Add C-language APIs for discovering SQL keywords used by SQLite: sqlite3_keyword_count(), sqlite3_keyword_name(), and sqlite3_keyword_check().
Add C-language APIs for dynamic strings based on the sqlite3_str object.
Enhance ALTER TABLE so that it recognizes "true" and "false" as valid arguments to DEFAULT.
Add the sorter-reference optimization as a compile-time option. Only available if compiled with SQLITE_ENABLE_SORTER_REFERENCES.
Improve the format of the EXPLAIN QUERY PLAN raw output, so that it gives better information about the query plan and about the relationships between the various components of the plan.
Added the SQLITE_DBCONFIG_RESET_DATABASE option to the sqlite3_db_config() API.
CLI Enhancements:
Automatically intercepts the raw EXPLAIN QUERY PLAN output and reformats it into an ASCII-art graph.
Lines that begin with "#" and that are not in the middle of an SQL statement are interpreted as comments.
Added the --append option to the ".backup" command.
Added the ".dbconfig" command.
Performance:
UPDATE avoids unnecessary low-level disk writes when the contents of the database file do not actually change. For example, "UPDATE t1 SET x=25 WHERE y=?" generates no extra disk I/O if the value in column x is already 25. Similarly, when doing UPDATE on records that span multiple pages, only the subset of pages that actually change are written to disk. This is a low-level performance optimization only and does not affect the behavior of TRIGGERs or other higher level SQL structures.
Queries that use ORDER BY and LIMIT now try to avoid computing rows that cannot possibly come in under the LIMIT. This can greatly improve performance of ORDER BY LIMIT queries, especially when the LIMIT is small relative to the number of unrestricted output rows.
The OR optimization is allowed to proceed even if the OR expression has also been converted into an IN expression. Uses of the OR optimization are now also more clearly shown in the EXPLAIN QUERY PLAN output.
The query planner is more aggressive about using automatic indexes for views and subqueries for which it is not possible to create a persistent index.
Make use of the one-pass UPDATE and DELETE query plans in the R-Tree extension where appropriate.
Performance improvements in the LEMON-generated parser.
Bug fixes:
For the right-hand table of a LEFT JOIN, compute the values of expressions directly rather than loading precomputed values out of an expression index as the expression index might not contain the correct value.
Do not attempt to use terms from the WHERE clause to enable indexed lookup of the right-hand table of a LEFT JOIN.
Fix a memory leak that can occur following a failure to open error in the CSV virtual table
Fix a long-standing problem wherein a corrupt schema on the sqlite_sequence table used by AUTOINCREMENT can lead to a crash.
Fix the json_each() function so that it returns valid results on its "fullkey" column when the input is a simple value rather than an array or object.
SQLite Release 3.23.1:
Fix two problems in the new LEFT JOIN strength reduction optimization.
Fix misbehavior of the FTS5 xBestIndex method.
Fix a harmless reference to an uninitialized virtual machine register.
Fix the CLI so that it builds with -DSQLITE_UNTESTABLE
Fix the eval.c extension so that it works with PRAGMA empty_result_callbacks=ON.
Fix the generate_series virtual table so that it correctly returns no rows if any of its constraints are NULL.
Performance enhancements in the parser.
SQLite Release 3.23.0:
Add the sqlite3_serialize() and sqlite3_deserialize() interfaces when the SQLITE_ENABLE_DESERIALIZE compile-time option is used.
Recognize TRUE and FALSE as constants. (For compatibility, if there exist columns named "true" or "false", then the identifiers refer to the columns rather than Boolean constants.)
Support operators IS TRUE, IS FALSE, IS NOT TRUE, and IS NOT FALSE.
Added the SQLITE_DBSTATUS_CACHE_SPILL option to sqlite3_db_status() for reporting the number of cache spills that have occurred.
The "alternate-form-2" flag ("!") on the built-in printf implementation now causes string substitutions to measure the width and precision in characters instead of bytes.
If the xColumn method in a virtual table implementation returns an error message using sqlite3_result_error() then give that error message preference over internally-generated messages.
Added the -A command-line option to the CLI to make it easier to manage SQLite Archive files.
Add support for INSERT OR REPLACE, INSERT OR IGNORE, and UPDATE OR REPLACE in the Zipfile virtual table.
Enhance the sqlite3changeset_apply() interface so that it is hardened against attacks from deliberately corrupted changeset objects.
Added the sqlite3_normalize() extension function.
Query optimizer enhancements:
Improve the omit-left-join optimization so that it works in cases where the right-hand table is UNIQUE but not necessarily NOT NULL.
Improve the push-down optimization so that it works for many LEFT JOINs.
Add the LEFT JOIN strength reduction optimization that converts a LEFT JOIN into an ordinary JOIN if there exist terms in the WHERE clause that would prevent the extra all-NULL row of the LEFT JOIN from appearing in the output set.
Avoid unnecessary writes to the sqlite_sequence table when an AUTOINCREMENT table is updated with an rowid that is less than the maximum.
Bug fixes:
Fix the parser to accept valid row value syntax.
Fix the query planner so that it takes into account dependencies in the arguments to table-valued functions in subexpressions in the WHERE clause.
Fix incorrect result with complex OR-connected WHERE and STAT4.
Fix potential corruption in indexes on expressions due to automatic datatype conversions.
Assertion fault in FTS4.
Incorrect result on the less-than operator in row values.
Always interpret non-zero floating-point values as TRUE, even if the integer part is zero.
Fix an issue in the fsdir(PATH) table-valued function to the fileio.c extension, that caused a segfault if the fsdir() table was used as the inner table of a join.
Issue an error rather instead of an assertion-fault or null-pointer dereference when the sqlite_master table is corrupted so that the sqlite_sequence table root page is really a btree-index page.
Fix the ANALYZE command so that it computes statistics on tables whose names begin with "sqlite".
Additional fixes for issues detected by OSSFuzz:
Fix a possible infinite loop on VACUUM for corrupt database files.
Disallow parameters in the WITH clause of triggers and views.
Fix a potential memory leak in row value processing.
Improve the performance of the replace() SQL function for cases where there are many substitutions on megabyte-sized strings, in an attempt to avoid OSSFuzz timeouts during testing.
Provide an appropriate error message when the sqlite_master table contains a CREATE TABLE AS statement. Formerly this caused either an assertion fault or null pointer dereference. Problem found by OSSFuzz on the GDAL project.
Incorrect assert() statement removed.
Fix a problem with using the LIKE optimization on an INTEGER PRIMARY KEY.
Release 3.22.0:
The output of sqlite3_trace_v2() now shows each individual SQL statement run within a trigger.
Add the ability to read from WAL mode databases even if the application lacks write permission on the database and its containing directory, as long as the -shm and -wal files exist in that directory.
Added the rtreecheck() scalar SQL function to the R-Tree extension.
Added the sqlite3_vtab_nochange() and sqlite3_value_nochange() interfaces to help virtual table implementations optimize UPDATE operations.
Added the sqlite3_vtab_collation() interface.
Added support for the "^" initial token syntax in FTS5.
New extensions:
The Zipfile virtual table can read and write a ZIP Archive.
Added the fsdir(PATH) table-valued function to the fileio.c extension, for listing the files in a directory.
The sqlite_btreeinfo eponymous virtual table for introspecting and estimating the sizes of the btrees in a database.
The Append VFS is a VFS shim that allows an SQLite database to be appended to some other file. This allows (for example) a database to be appended to an executable that then opens and reads the database.
Query planner enhancements:
The optimization that uses an index to quickly compute an aggregate min() or max() is extended to work with indexes on expressions.
The decision of whether to implement a FROM-clause subquery as a co-routine or using query flattening now considers whether the result set of the outer query is "complex" (if it contains functions or expression subqueries). A complex result set biases the decision toward the use of co-routines.
The planner avoids query plans that use indexes with unknown collating functions.
The planner omits unused LEFT JOINs even if they are not the right-most joins of a query.
Other performance optimizations:
A smaller and faster implementation of text to floating-point conversion subroutine: sqlite3AtoF().
The Lemon parser generator creates a faster parser.
Use the strcspn() C-library routine to speed up the LIKE and GLOB operators.
Improvements to the command-line shell:
The ".schema" command shows the structure of virtual tables.
Added support for reading and writing SQL Archive files using the .archive command.
Added the experimental .expert command
Added the ".eqp trigger" variant of the ".eqp" command
Enhance the ".lint fkey-indexes" command so that it works with WITHOUT ROWID tables.
If the filename argument to the shell is a ZIP archive rather than an SQLite database, then the shell automatically opens that ZIP archive using the Zipfile virtual table.
Added the edit() SQL function.
Added the .excel command to simplify exporting database content to a spreadsheet.
Databases are opened using Append VFS when the --append flag is used on the command line or with the .open command.
Enhance the SQLITE_ENABLE_UPDATE_DELETE_LIMIT compile-time option so that it works for WITHOUT ROWID tables.
Provide the sqlite_offset(X) SQL function that returns the byte offset into the database file to the beginning of the record holding value X, when compiling with -DSQLITE_ENABLE_OFFSET_SQL_FUNC.
Bug fixes
Release 3.21.0:
Take advantage of the atomic-write capabilities in the F2FS filesystem when available, for greatly reduced transaction overhead. This currently requires the SQLITE_ENABLE_BATCH_ATOMIC_WRITE compile-time option.
Allow ATTACH and DETACH commands to work inside of a transaction.
Allow WITHOUT ROWID virtual tables to be writable if the PRIMARY KEY contains exactly one column.
The "fsync()" that occurs after the header is written in a WAL reset now uses the sync settings for checkpoints. This means it will use a "fullfsync" on macs if PRAGMA checkpoint_fullfsync set on.
The sqlite3_sourceid() function tries to detect if the source code has been modified from what is checked into version control and if there are modifications, the last four characters of the version hash are shown as "alt1" or "alt2". The objective is to detect accidental and/or careless edits. A forger can subvert this feature.
Improved de-quoting of column names for CREATE TABLE AS statements with an aggregate query on the right-hand side.
Fewer "stat()" system calls issued by the unix VFS.
Enhanced the LIKE optimization so that it works with an ESCAPE clause.
Enhanced PRAGMA integrity_check and PRAGMA quick_check to detect obscure row corruption that they were formerly missing. Also update both pragmas so that they return error text rather than SQLITE_CORRUPT when encountering corruption in records.
The query planner now prefers to implement FROM-clause subqueries using co-routines rather using the query flattener optimization. Support for the use of co-routines for subqueries may no longer be disabled.
Pass information about !=, IS, IS NOT, NOT NULL, and IS NULL constraints into the xBestIndex method of virtual tables.
Enhanced the CSV virtual table so that it accepts the last row of input if the final new-line character is missing.
Remove the rarely-used "scratch" memory allocator. Replace it with the SQLITE_CONFIG_SMALL_MALLOC configuration setting that gives SQLite a hint that large memory allocations should be avoided when possible.
Added the swarm virtual table to the existing union virtual table extension.
Added the sqlite_dbpage virtual table for providing direct access to pages of the database file. The source code is built into the amalgamation and is activated using the -DSQLITE_ENABLE_DBPAGE_VTAB compile-time option.
Add a new type of fts5vocab virtual table - "instance" - that provides direct access to an FTS5 full-text index at the lowest possible level.
Remove a call to rand_s() in the Windows VFS since it was causing problems in Firefox on some older laptops.
The src/shell.c source code to the command-line shell is no longer under version control. That file is now generated as part of the build process.
Miscellaneous microoptimizations reduce CPU usage by about 2.1%.
Bug fixes
This package uses sqlite3/Makefile.version to set the package
version, but the checksum was for the distfile of the previous
version. Update the checksum to the correct distfile and fix the
resulting PLIST.
Fix more bugs in the LEFT JOIN flattening optimization. Ticket 7fde638e94287d2c.
Changes carried forward from version 3.19.1 (2017-05-24):
Fix a bug in the LEFT JOIN flattening optimization. Ticket cad1ab4cb7b0fc.
Remove a surplus semicolon that was causing problems for older versions of MSVC.
---------------------
The SQLITE_READ authorizer callback is invoked once with a column name that is an empty string for every table referenced in a query from which no columns are extracted.
When using an index on an expression, try to use expression values already available in the index, rather than loading the original columns and recomputing the expression.
Enhance the flattening optimization so that it is able to flatten views on the right-hand side of a LEFT JOIN.
Use replace() instead of char() for escaping newline and carriage-return characters embedded in strings in the .dump output from the command-line shell.
Avoid unnecessary foreign key processing in UPDATE statements that do not touch the columns that are constrained by the foreign keys.
On a DISTINCT query that uses an index, try to skip ahead to the next distinct entry using the index rather than stepping through rows, when an appropriate index is available.
Avoid unnecessary invalidation of sqlite3_blob handles when making changes to unrelated tables.
Transfer any terms of the HAVING clause that use only columns mentioned in the GROUP BY clause over to the WHERE clause for faster processing.
Reuse the same materialization of a VIEW if that VIEW appears more than once in the same query.
Enhance PRAGMA integrity_check so that it identifies tables that have two or more rows with the same rowid.
Enhance the FTS5 query syntax so that column filters may be applied to arbitrary expressions.
Enhance the json_extract() function to cache and reuse parses of JSON input text.
Added the anycollseq.c loadable extension that allows a generic SQLite database connection to read a schema that contains unknown and/or application-specific collating sequences.
Bug Fixes:
----------
Fix a problem in REPLACE that can result in a corrupt database containing two or more rows with the same rowid.
Fix a problem in PRAGMA integrity_check that was causing a subsequent VACUUM to behave suboptimally.
Fix the PRAGMA foreign_key_check command so that it works correctly with foreign keys on WITHOUT ROWID tables.
Disallow leading zeros in numeric constants in JSON.
Disallow control characters inside of strings in JSON.
Limit the depth of recursion for JSON objects and arrays in order to avoid excess stack usage in the recursive descent parser.
Added the PRAGMA optimize command
The SQLite version identifier returned by the sqlite_source_id() SQL function and the sqlite3_sourceid() C API and found in the SQLITE_SOURCE_ID macro is now a 64-digit SHA3-256 hash instead of a 40-digit SHA1 hash.
Added the json_patch() SQL function to the JSON1 extension.
Enhance the LIKE optimization so that it works for arbitrary expressions on the left-hand side as long as the LIKE pattern on the right-hand side does not begin with a digit or minus sign.
Added the sqlite3_set_last_insert_rowid() interface and use the new interface in the FTS3, FTS4, and FTS5 extensions to ensure that the sqlite3_last_insert_rowid() interface always returns reasonable values.
Enhance PRAGMA integrity_check and PRAGMA quick_check so that they verify CHECK constraints.
Enhance the query plans for joins to detect empty tables early and halt without doing unnecessary work.
Enhance the sqlite3_mprintf() family of interfaces and the printf SQL function to put comma separators at the thousands marks for integers, if the "," format modifier is used in between the "%" and the "d" (example: "%,d").
Added the -DSQLITE_MAX_MEMORY=N compile-time option.
Added the .sha3sum dot-command and the .selftest dot-command to the command-line shell
Begin enforcing SQLITE_LIMIT_VDBE_OP. This can be used, for example, to prevent excessively large prepared statements in systems that accept SQL queries from untrusted users.
Various performance improvements.
Bug Fixes:
----------
Ensure that indexed expressions with collating sequences are handled correctly. Fix for ticket eb703ba7b50c1a5.
Fix a bug in the 'start of ...' modifiers for the date and time functions. Ticket 6097cb92745327a1
Fix a potential segfault in complex recursive triggers, resulting from a bug in the OP_Once opcode introduced as part of a performance optimization in version 3.15.0. Ticket 06796225f59c057c
In the RBU extension, add extra sync operations to avoid the possibility of corruption following a power failure.
The sqlite3_trace_v2() output for nested SQL statements should always begin with a "--" comment marker.
* Approximately 25% better performance from the R-Tree extension.
* Uses compiler built-ins (ex: __builtin_bswap32() or _byteswap_ulong()) for byteswapping when available.
* Uses the sqlite3_blob key/value access object instead of SQL for pulling content out of R-Tree nodes
* Other miscellaneous enhancements such as loop unrolling.
* Add the SQLITE_DEFAULT_LOOKASIDE compile-time option.
* Increase the default lookaside size from 512,125 to 1200,100 as this provides better performance while only adding 56KB of extra memory per connection. Memory-sensitive applications can restore the old default at compile-time, start-time, or run-time.
* Use compiler built-ins __builtin_sub_overflow(), __builtin_add_overflow(), and __builtin_mul_overflow() when available. (All compiler built-ins can be omitted with the SQLITE_DISABLE_INTRINSIC compile-time option.)
* Added the SQLITE_ENABLE_NULL_TRIM compile-time option, which can result in significantly smaller database files for some applications, at the risk of being incompatible with older versions of SQLite.
* Change SQLITE_DEFAULT_PCACHE_INITSZ from 100 to 20, for improved performance.
* Added the SQLITE_UINT64_TYPE compile-time option as an analog to SQLITE_INT64_TYPE.
* Perform some UPDATE operations in a single pass instead of in two passes.
* Enhance the session extension to support WITHOUT ROWID tables.
* Fixed performance problems and potential stack overflows when creating views from multi-row VALUES clauses with hundreds of thousands of rows.
* Added the sha1.c extension.
* In the command-line shell, enhance the ".mode" command so that it restores the default column and row separators for modes "line", "list", "column", and "tcl".
* Enhance the SQLITE_DIRECT_OVERFLOW_READ option so that it works in WAL mode as long as the pages being read are not in the WAL file.
* Enhance the LEMON parser generator so that it can store the parser object as a stack variable rather than allocating space from the heap and make use of that enhancement in the amalgamation.
* Other performance improvements. Uses about 6.5% fewer CPU cycles.
Bug Fixes:
* Throw an error if the ON clause of a LEFT JOIN references tables to the right of the ON clause. This is the same behavior as PostgreSQL. Formerly, SQLite silently converted the LEFT JOIN into an INNER JOIN.
* Use the correct affinity for columns of automatic indexes.
* Ensure that the sqlite3_blob_reopen() interface can correctly handle short rows.
Fix the REPLACE statement for WITHOUT ROWID tables that lack secondary indexes so that it works correctly with triggers and foreign keys. This was a new bug caused by performance optimizations added in version 3.16.0. Ticket 30027b613b4
Fix the sqlite3_value_text() interface so that it correctly translates content generated by zeroblob() into a string of all 0x00 characters. This is a long-standing issue discovered after the 3.16.1 release by OSS-Fuzz
Fix the bytecode generator to deal with a subquery in the FROM clause that is itself a UNION ALL where one side of the UNION ALL is a view that contains an ORDER BY. This is a long-standing issue that was discovered after the release of 3.16.1. See ticket 190c2507.
Adjust the sqlite3_column_count() API so it more often returns the same values for PRAGMA statements as it did in prior releases, to minimize disruption to applications that might be using that interface in unexpected ways.
Fix a bug concerning the use of row values within triggers (see ticket 8c9458e7) that was in version 3.15.0 but was not reported until moments after the 3.16.0 release was published.
Uses 9% fewer CPU cycles. (See the CPU performance measurement report for details on how the this performance increase was computed.)
Added experimental support for PRAGMA functions.
Added the SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE option to sqlite3_db_config().
Enhance the date and time functions so that the 'unixepoch' modifier works for the full span of supported dates.
Changed the default configuration of the lookaside memory allocator from 500 slots of 128 bytes each into 125 slots of 512 bytes each.
Enhanced "WHERE x NOT NULL" partial indexes so that they are usable if the "x" column appears in a LIKE or GLOB operator.
Enhanced sqlite3_interrupt() so that it interrupts checkpoint operations that are in process.
Enhanced the LIKE and GLOB matching algorithm to be faster for cases when the pattern contains multiple wildcards.
Added the SQLITE_FCNTL_WIN32_GET_HANDLE file control opcode.
Added ".mode quote" to the command-line shell.
Added ".lint fkey-indexes" to the command-line shell.
Added the .imposter dot-command to the command-line shell.
Added the remember(V,PTR) SQL function as a loadable extension.
Rename the SQLITE_OMIT_BUILTIN_TEST compile-time option to SQLITE_UNTESTABLE to better reflect the implications of using it.
Bug Fixes:
Fix a long-standing bug in the query planner that caused incorrect results on a LEFT JOIN where the left-hand table is a subquery and the join constraint is a bare column name coming from the left-hand subquery. Ticket 2df0107b.
Correctly handle the integer literal -0x8000000000000000 in the query planner.
* Added SQLITE_FCNTL_WIN32_GET_HANDLE file control opcode.
Bug Fixes:
* Fix the VACUUM command so that it spills excess content to disk rather than holding everything in memory, and possible causing an out-of-memory error for larger database files. This fixes an issue introduced by version 3.15.0.
* Fix a case (present since 3.8.0 - 2013-08-26) where OR-connected terms in the ON clause of a LEFT JOIN might cause incorrect results.
* Fix a case where the use of row values in the ON clause of a LEFT JOIN might cause incorrect results.
Added support for row values.
Allow deterministic SQL functions in the WHERE clause of a partial index.
Added the "modeof=filename" URI parameter on the unix VFS
Added support for SQLITE_DBCONFIG_MAINDBNAME.
Added the ability to VACUUM an ATTACH-ed database.
Enhancements to the command-line shell:
Add the ".testcase" and ".check" dot-commands.
Added the --new option to the ".open" dot-command, causing any prior content in the database to be purged prior to opening.
Enhance the fts5vocab virtual table to handle "ORDER BY term" efficiently.
Miscellaneous micro-optimizations reduce CPU usage by more than 7% on common workloads. Most optimization in this release has been on the front-end (sqlite3_prepare_v2()).
Bug Fixes:
The multiply operator now correctly detects 64-bit integer overflow and promotes to floating point in all corner-cases.
Correct handling of columns with redundant unique indexes when those columns are used on the LHS of an IN operator.
Skip NULL entries on range queries in indexes on expressions.
Ensure that the AUTOINCREMENT counters in the sqlite_sequence table are initialized doing "Xfer Optimization" on "INSERT ... SELECT" statements.
Make sure the ORDER BY LIMIT optimization (from check-in 559733b09e) works with IN operators on INTEGER PRIMARY KEYs.
* Improved support for using the STDCALL calling convention in winsqlite3.dll.
* Fix the sqlite3_trace_v2() interface so that it is disabled if either the callback or the mask arguments are zero, in accordance with the documentation.
* Fix commenting errors and improve the comments generated on EXPLAIN listings when the -DSQLITE_ENABLE_EXPLAIN_COMMENTS compile-time option is used.
* Fix the ".read" command in the command-line shell so that it understands that its input is not interactive.
* Correct affinity computations for a SELECT on the RHS of an IN operator.
* The ORDER BY LIMIT optimization is not valid unless the inner-most IN operator loop is actually used by the query plan. Fix for ticket 0c4df46116e90f92.
* Fix an internal code generator problem that was causing some DELETE operations to no-op.
* A performance enhancement to the page-cache "truncate" operation reduces COMMIT time by dozens of milliseconds on systems with a large page cache.
* Fix to the --rbu option of sqldiff.
Added support for WITHOUT ROWID virtual tables.
Improved the query planner so that the OR optimization can be used on virtual tables even if one or more of the disjuncts use the LIKE, GLOB, REGEXP, MATCH operators.
Added the CSV virtual table for reading RFC 4180 formatted comma-separated value files.
Added the carray() table-valued function extension.
Enabled persistent loadable extensions using the new SQLITE_OK_LOAD_PERMANENTLY return code from the extension entry point.
Added the SQLITE_DBSTATUS_CACHE_USED_SHARED option to sqlite3_db_status().
Add the vfsstat.c loadable extension - a VFS shim that measures I/O together with an eponymous virtual table that provides access to the measurements.
Improved algorithm for running queries with both an ORDER BY and a LIMIT where only the inner-most loop naturally generates rows in the correct order.
Enhancements to Lemon parser generator, so that it generates a faster parser.
The PRAGMA compile_options command now attempts to show the version number of the compiler that generated the library.
Enhance PRAGMA table_info so that it provides information about eponymous virtual tables.
Added the "win32-none" VFS, analogous to the "unix-none" VFS, that works like the default "win32" VFS except that it ignores all file locks.
The query planner uses a full scan of a partial index instead of a full scan of the main table, in cases where that makes sense.
Allow table-valued functions to appear on the right-hand side of an IN operator.
Created the dbhash.exe command-line utility.
Added two new C-language interfaces: sqlite3_expanded_sql() and sqlite3_trace_v2(). These new interfaces subsume the functions of sqlite3_trace() and sqlite3_profile() which are now deprecated.
Added the json_quote() SQL function to the json1 extension.
Disable the authorizer callback while reparsing the schema.
Added the SQLITE_ENABLE_UNKNOWN_SQL_FUNCTION compile-time option and turned that option on by default when building the command-line shell.
Bug Fixes:
Fix the ALTER TABLE command so that it does not corrupt descending indexes when adding a column to a legacy file format database.
Fix a NULL-pointer dereference/crash that could occurs when a transitive WHERE clause references a non-existent collating sequence.
Improved the cost estimation for an index scan which includes a WHERE clause that can be partially or fully evaluated using columns in the index and without having to do a table lookup. This fixes a performance regression that occurred for some obscure queries following the ORDER BY LIMIT optimization introduced in version 3.12.0.
Postpone I/O associated with TEMP files for as long as possible, with the hope that the I/O can ultimately be avoided completely.
Merged the session extension into trunk.
Added the ".auth ON|OFF" command to the command-line shell.
Added the "--indent" option to the ".schema" and ".fullschema" commands of the command-line shell, to turn on pretty-printing.
Added the ".eqp full" option to the command-line shell, that does both EXPLAIN and EXPLAIN QUERY PLAN on each statement that is evaluated.
Improved unicode filename handling in the command-line shell on Windows.
Improved resistance against goofy query planner decisions caused by incomplete or incorrect modifications to the sqlite_stat1 table by the application.
Added the sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION) interface which allows the sqlite3_load_extension() C-API to be enabled while keeping the load_extension() SQL function disabled for security.
Change the temporary directory search algorithm on Unix to allow directories with write and execute permission, but without read permission, to serve as temporary directories. Apply this same standard to the "." fallback directory.
Bug Fixes:
----------
Fix a problem with the multi-row one-pass DELETE optimization that was causing it to compute incorrect answers with a self-referential subquery in the WHERE clause. Fix for ticket dc6ebeda9396087
Fix a possible segfault with DELETE when table is a rowid table with an INTEGER PRIMARY KEY and the WHERE clause contains a OR and the table has one or more indexes that are able to trigger the OR optimization, but none of the indexes reference any table columns other than the INTEGER PRIMARY KEY. Ticket 16c9801ceba49.
When checking for the WHERE-clause push-down optimization, verify that all terms of the compound inner SELECT are non-aggregate, not just the last term. Fix for ticket f7f8c97e97597.
Fix a locking race condition in Windows that can occur when two or more processes attempt to recover the same hot journal at the same time.
Fix a backwards compatibility problem in version 3.12.0 and 3.12.1: Columns declared as "INTEGER" PRIMARY KEY (with quotes around the datatype keyword) where not being recognized as an INTEGER PRIMARY KEY, which resulted in an incompatible database file.
Fix a bug (present since version 3.9.0) that can cause the DELETE operation to miss rows if PRAGMA reverse_unordered_selects is turned on.
Fix a bug in the code generator that can causes incorrect results if two or more virtual tables are joined and the virtual table used in outer loop of the join has an IN operator constraint.
Correctly interpret negative "PRAGMA cache_size" values when determining the cache size used for sorting large amounts of data.
Fix a boundary condition error introduced by version 3.12.0 that can result in a crash during heavy SAVEPOINT usage.
Fix views so that they inherit column datatypes from the table that they are defined against, when possible.
Fix the query planner so that IS and IS NULL operators are able to drive an index on a LEFT OUTER JOIN.
Potentially Disruptive Change:
------------------------------
The SQLITE_DEFAULT_PAGE_SIZE is increased from 1024 to 4096. The SQLITE_DEFAULT_CACHE_SIZE is changed from 2000 to -2000 so the same amount of cache memory is used by default. See the application note on the version 3.12.0 page size change for further information.
Performance enhancements:
-------------------------
Enhancements to the Lemon parser generator so that it creates a smaller and faster SQL parser.
Only create master journal files if two or more attached databases are all modified, do not have PRAGMA synchronous set to OFF, and do not have the journal_mode set to OFF, MEMORY, or WAL.
Only create statement journal files when their size exceeds a threshold. Otherwise the journal is held in memory and no I/O occurs. The threshold can be configured at compile-time using SQLITE_STMTJRNL_SPILL or at start-time using sqlite3_config(SQLITE_CONFIG_STMTJRNL_SPILL).
The query planner is able to optimize IN operators on virtual tables even if the xBestIndex method does not set the sqlite3_index_constraint_usage.omit flag of the virtual table column to the left of the IN operator.
The query planner now does a better job of optimizing virtual table accesses in a 3-way or higher join where constraints on the virtual table are split across two or more other tables of the join.
More efficient handling of application-defined SQL functions, especially in cases where the application defines hundreds or thousands of custom functions.
The query planner considers the LIMIT clause when estimating the cost of ORDER BY.
The configure script (on unix) automatically detects pread() and pwrite() and sets compile-time options to use those OS interfaces if they are available.
Reduce the amount of memory needed to hold the schema.
Other miscellaneous micro-optimizations for improved performance and reduced memory usage.
New Features:
-------------
Added the SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER option to sqlite3_db_config() which allows the two-argument version of the fts3_tokenizer() SQL function to be enabled or disabled at run-time.
Added the sqlite3rbu_bp_progress() interface to the RBU extension.
The PRAGMA defer_foreign_keys=ON statement now also disables RESTRICT actions on foreign key.
Added the sqlite3_system_errno() interface.
Added the SQLITE_DEFAULT_SYNCHRONOUS and SQLITE_DEFAULT_WAL_SYNCHRONOUS compile-time options. The SQLITE_DEFAULT_SYNCHRONOUS compile-time option replaces the SQLITE_EXTRA_DURABLE option, which is no longer supported.
Enhanced the ".stats" command in the command-line shell to show more information about I/O performance obtained from /proc, when available.
Bug fixes:
----------
Make sure the sqlite3_set_auxdata() values from multiple triggers within a single statement do not interfere with one another.
Fix the code generator for expressions of the form "x IN (SELECT...)" where the SELECT statement on the RHS is a correlated subquery.
Fix a harmless TSAN warning associated with the sqlite3_db_readonly() interface.
Improvements to the Makefiles and build scripts used by VisualStudio.
Fix an FTS5 issue in which the 'optimize' command could cause index corruption.
Fix a buffer overread that might occur if FTS5 is used to query a corrupt database file.
Increase the maximum "scope" value for the spellfix1 extension from 6 to 30.
General improvements:
Enhanced WAL mode so that it works efficiently with transactions that are larger than the cache_size.
Added the FTS5 detail option.
Added the "EXTRA" option to PRAGMA synchronous that does a sync of the containing directory when a rollback journal is unlinked in DELETE mode, for better durability. The SQLITE_EXTRA_DURABLE compile-time option enables PRAGMA synchronous=EXTRA by default.
Enhanced the query planner so that it is able to use a covering index as part of the OR optimization.
Avoid recomputing NOT NULL and CHECK constraints on unchanged columns in UPDATE statement.
Many micro-optimizations, resulting in a library that is faster than the previous release.
Enhancements to the command-line shell:
By default, the shell is now in "auto-explain" mode. The output of EXPLAIN commands is automatically formatted.
Added the ".vfslist" dot-command.
The SQLITE_ENABLE_EXPLAIN_COMMENTS compile-time option is now turned on by default in the standard builds.
Enhancements to the TCL Interface:
If a database connection is opened with the "-uri 1" option, then URI filenames are honored by the "backup" and "restore" commands.
Added the "-sourceid" option to the "sqlite3" command.
Makefile improvements:
Improved pthreads detection in configure scripts.
Add the ability to do MSVC Windows builds from the amalgamation tarball.
Bug fixes
Fix an issue with incorrect sharing of VDBE temporary registers between co-routines that could cause incorrect query results in obscure cases. Ticket d06a25c84454a.
Fix a problem in the sqlite3_result_subtype() interface that could cause problems for the json1 extension under obscure circumstances. Fix for ticket f45ac567eaa9f9.
Escape control characters in JSON strings. Fix for ticket ad2559db380abf8.
Reenable the xCurrentTime and xGetLastError methods in the built-in unix VFSes as long as SQLITE_OMIT_DEPRECATED is not defined.
Backwards Compatibility:
Because of continuing security concerns, the two-argument version of of the seldom-used and little-known fts3_tokenizer() function is disabled unless SQLite is compiled with the SQLITE_ENABLE_FTS3_TOKENIZER.
* Version 3.10.0 introduced a case-folding bug in the LIKE operator which is fixed by this patch release.
* Fix a use-after-free that can occur when SQLite is compiled with -DSQLITE_HAS_CODEC.
* Fix the build so that it works with -DSQLITE_OMIT_WAL.
* Fix the configure script for the amalgamation so that the --readline option works again on Raspberry PIs.
* Add the SQLITE_FCNTL_JOURNAL_POINTER file control.
* Fix a 16-month-old bug in the query planner that could generate incorrect results when a scalar subquery attempts to use the block sorting optimization.
General improvements:
---------------------
Added support for LIKE, GLOB, and REGEXP operators on virtual tables.
Added the colUsed field to sqlite3_index_info for use by the sqlite3_module.xBestIndex method.
Enhance the PRAGMA cache_spill statement to accept a 32-bit integer parameter which is the threshold below which cache spilling is prohibited.
On unix, if a symlink to a database file is opened, then the corresponding journal files are based on the actual filename, not the symlink name.
Added the "--transaction" option to sqldiff.
Added the sqlite3_db_cacheflush() interface.
Added the sqlite3_strlike() interface.
When using memory-mapped I/O map the database file read-only so that stray pointers and/or array overruns in the application cannot accidently modify the database file.
Added the experimental sqlite3_snapshot_get(), sqlite3_snapshot_open(), and sqlite3_snapshot_free() interfaces. These are subject to change or removal in a subsequent release.
Enhance the 'utc' modifier in the date and time functions so that it is a no-op if the date/time is known to already be in UTC. (This is not a compatibility break since the behavior has long been documented as "undefined" in that case.)
Added the json_group_array() and json_group_object() SQL functions in the json extension.
Added the SQLITE_LIKE_DOESNT_MATCH_BLOBS compile-time option.
Many small performance optimizations.
Portability enhancements:
-------------------------
Work around a sign-exension bug in the optimizer of the HP C compiler on HP/UX. (details)
Enhancements to the command-line shell:
---------------------------------------
Added the ".changes ON|OFF" and ".vfsinfo" dot-commands.
Translate between MBCS and UTF8 when running in cmd.exe on Windows.
Enhancements to makefiles:
--------------------------
Added the --enable-editline and --enable-static-shell options to the various autoconf-generated configure scripts.
Omit all use of "awk" in the makefiles, to make building easier for MSVC users.
Important fixes:
----------------
Fix inconsistent integer to floating-point comparison operations that could result in a corrupt index if the index is created on a table column that contains both large integers and floating point values of similar magnitude. Ticket 38a97a87a6.
Fix an infinite-loop in the query planner that could occur on malformed common table expressions.
Various bug fixes in the sqldiff tool.
* Fix the schema parser so that it interprets certain (obscure and ill-formed) CREATE TABLE statements the same as legacy.
* Fix a query planner problem that could result in an incorrect answer due to the use of automatic indexing in subqueries in the FROM clause of a correlated scalar subqueries.
Problems found with existing distfiles:
distfiles/D6.data.ros.gz
distfiles/cstore0.2.tar.gz
distfiles/data4.tar.gz
distfiles/sphinx-2.2.7-release.tar.gz
No changes made to the cstore or mariadb55-client distinfo files.
Otherwise, existing SHA1 digests verified and found to be the same on
the machine holding the existing distfiles (morden). All existing
SHA1 digests retained for now as an audit trail.
Fix the json1 extension so that it does not recognize ASCII form-feed as a whitespace character, in order to comply with RFC-7159.
Add a few #ifdef and build script changes to address compilation issues that appeared after the 3.9.0 release.
Policy Changes:
* The version numbering conventions for SQLite are revised to use the emerging standard of semantic versioning.
New Features And Enhancements:
* Added the json1 extension module in the source tree, and in the amalgamation. Enable support using the SQLITE_ENABLE_JSON1 compile-time option.
* Added Full Text Search version 5 (FTS5) to the amalgamation, enabled using SQLITE_ENABLE_FTS5. FTS5 will be considered "experimental" (subject to incompatible changes) for at least one more release cycle.
* The CREATE VIEW statement now accepts an optional list of column names following the view name.
* Added support for indexes on expressions.
* Added support for table-valued functions in the FROM clause of a SELECT statement.
* Added support for eponymous virtual tables.
* A VIEW may now reference undefined tables and functions when initially created. Missing tables and functions are reported when the VIEW is used in a query.
* Added the sqlite3_value_subtype() and sqlite3_result_subtype() interfaced (used by the json1 extension).
* The query planner is now able to use partial indexes that contain AND-connected terms in the WHERE clause.
* The sqlite3_analyzer.exe utility is updated to report the depth of each btree and to show the average fanout for indexes and WITHOUT ROWID tables.
* Enhanced the dbstat virtual table so that it can be used as a table-valued function where the argument is the schema to be analyzed.
Other changes:
* The sqlite3_memory_alarm() interface, which has been deprecated and undocumented for 8 years, is changed into a no-op.
Important fixes:
* Fixed a critical bug in the SQLite Encryption Extension that could cause the database to become unreadable and unrecoverable if a VACUUM command changed the size of the encryption nonce.
* Added a memory barrier in the implementation of sqlite3_initialize() to help ensure that it is thread-safe.
* Fix the OR optimization so that it always ignores subplans that do not use an index.
* Do not apply the WHERE-clause pushdown optimization on terms that originate in the ON or USING clause of a LEFT JOIN.
Restore an undocumented side-effect of PRAGMA cache_size: force the database schema to be parsed if the database has not been previously accessed.
Fix a long-standing problem in sqlite3_changes() for WITHOUT ROWID tables that was reported a few hours after the 3.8.11 release.
Added the experimental RBU extension. Note that this extension is experimental and subject to change in incompatible ways.
Added the experimental FTS5 extension. Note that this extension is experimental and subject to change in incompatible ways.
Added the sqlite3_value_dup() and sqlite3_value_free() interfaces.
Enhance the spellfix1 extension to support ON CONFLICT clauses.
The IS operator is now able to drive indexes.
Enhance the query planner to permit automatic indexing on FROM-clause subqueries that are implemented by co-routine.
Disallow the use of "rowid" in common table expressions.
Added the PRAGMA cell_size_check command for better and earlier detection of database file corruption.
Added the matchinfo 'b' flag to the matchinfo() function in FTS3.
Improved fuzz-testing of database files, with fixes for problems found.
Add the fuzzcheck test program and automatically run this program using both SQL and database test cases on "make test".
Added the SQLITE_MUTEX_STATIC_VFS1 static mutex and use it in the Windows VFS.
The sqlite3_profile() callback is invoked (by sqlite3_reset() or sqlite3_finalize()) for statements that did not run to completion.
Enhance the page cache so that it can preallocate a block of memory to use for the initial set page cache lines. Set the default preallocation to 100 pages. Yields about a 5% performance increase on common workloads.
Miscellaneous micro-optimizations result in 22.3% more work for the same number of CPU cycles relative to the previous release. SQLite now runs twice as fast as version 3.8.0 and three times as fast as version 3.3.9. (Measured using cachegrind on the speedtest1.c workload on Ubuntu 14.04 x64 with gcc 4.8.2 and -Os. Your performance may vary.)
Added the sqlite3_result_zeroblob64() and sqlite3_bind_zeroblob64() interfaces.
Fix an index corruption issue introduced by version 3.8.7. An index with a TEXT key can be corrupted by an INSERT into the corresponding table if the table has two nested triggers that convert the key value to INTEGER and back to TEXT again.
Make sqlite3_compileoption_used() responsive to the SQLITE_ENABLE_DBSTAT_VTAB compile-time option.
Fix a harmless warning in the command-line shell on some versions of MSVC.
Fix minor issues with the dbstat virtual table.
Added the sqldiff.exe utility program for computing the differences between two SQLite database files.
Added the y format string to the matchinfo() function of FTS3.
Performance improvements for ORDER BY, VACUUM, CREATE INDEX, PRAGMA integrity_check, and PRAGMA quick_check.
Fix many obscure problems discovered while SQL fuzzing.
Identify all methods for important objects in the interface documentation. (example)
Made the American Fuzzy Lop fuzzer a standard part of SQLite's testing strategy.
Add the ".binary" and ".limits" commands to the command-line shell.
Make the "dbstat" virtual table part of standard builds when compiled with the SQLITE_ENABLE_DBSTAT_VTAB option.
Add VxWorks-7 as an officially supported and tested platform.
Added the sqlite3_status64() interface.
Fix memory size tracking so that it works even if SQLite uses more than 2GiB of memory.
Added the PRAGMA index_xinfo command.
Fix a potential 32-bit integer overflow problem in the sqlite3_blob_read() and sqlite3_blob_write() interfaces.
Ensure that prepared statements automatically reset on extended error codes of SQLITE_BUSY and SQLITE_LOCKED even when compiled using SQLITE_OMIT_AUTORESET.
Correct miscounts in the sqlite3_analyzer.exe utility related to WITHOUT ROWID tables.
Added the ".dbinfo" command to the command-line shell.
Improve the performance of fts3/4 queries that use the OR operator and at least one auxiliary fts function.
Fix a bug in the fts3 snippet() function causing it to omit leading separator characters from snippets that begin with the first token in a column.
Fix a bug (ticket 2326c258d02ead33) that can lead to incorrect results if the qualifying constraint of a partial index appears in the ON clause of a LEFT JOIN.
Added the ability to link against the "linenoise" command-line editing library in unix builds of the command-line shell.
* Fix a bug in the sorting logic, present since version 3.8.4, that can cause output to appear in the wrong order on queries that contains an ORDER BY clause, a LIMIT clause, and that have approximately 60 or more columns in the result set.
New Features:
* Added the PRAGMA data_version command that can be used to determine if a database file has been modified by another process.
* Added the SQLITE_CHECKPOINT_TRUNCATE option to the sqlite3_wal_checkpoint_v2() interface, with corresponding enhancements to PRAGMA wal_checkpoint.
* Added the sqlite3_stmt_scanstatus() interface, available only when compiled with SQLITE_ENABLE_STMT_SCANSTATUS.
* The sqlite3_table_column_metadata() is enhanced to work correctly on WITHOUT ROWID tables and to check for the existence of a a table if the column name parameter is NULL. The interface is now also included in the build by default, without requiring the SQLITE_ENABLE_COLUMN_METADATA compile-time option.
* Added the SQLITE_ENABLE_API_ARMOR compile-time option.
* Added the SQLITE_REVERSE_UNORDERED_SELECTS compile-time option.
* Added the SQLITE_SORTER_PMASZ compile-time option and SQLITE_CONFIG_PMASZ start-time option.
* Added the SQLITE_CONFIG_PCACHE_HDRSZ option to sqlite3_config() which makes it easier for applications to determine the appropriate amount of memory for use with SQLITE_CONFIG_PAGECACHE.
* The number of rows in a VALUES clause is no longer limited by SQLITE_LIMIT_COMPOUND_SELECT.
* Added the eval.c loadable extension that implements an eval() SQL function that will recursively evaluate SQL.
Performance Enhancements:
* Reduce the number of memcpy() operations involved in balancing a b-tree, for 3.2% overall performance boost.
* Improvements to cost estimates for the skip-scan optimization.
* The automatic indexing optimization is now capable of generating a partial index if that is appropriate.
Bug fixes:
* Ensure durability following a power loss with "PRAGMA journal_mode=TRUNCATE" by calling fsync() right after truncating the journal file.
* The query planner now recognizes that any column in the right-hand table of a LEFT JOIN can be NULL, even if that column has a NOT NULL constraint. Avoid trying to optimize out NULL tests in those cases.
* Make sure ORDER BY puts rows in ascending order even if the DISTINCT operator is implemented using a descending index.
* Fix data races that might occur under stress when running with many threads in shared cache mode where some of the threads are opening and closing connections.
* Fix obscure crash bugs found by american fuzzy lop.
* Work around a GCC optimizer bug (for gcc 4.2.1 on MacOS 10.7) that caused the R-Tree extension to compute incorrect results when compiled with -O3.
Other changes:
* Disable the use of the strchrnul() C-library routine unless it is specifically enabled using the -DHAVE_STRCHRNULL compile-time option.
* Improvements to the effectiveness and accuracy of the likelihood(), likely(), and unlikely() SQL hint functions.
* Bug fix: Ensure the cached KeyInfo objects (an internal abstraction not visible to the application) do not go stale when operating in shared cache mode and frequently closing and reopening some database connections while leaving other database connections on the same shared cache open continuously.
* Bug fix: Recognize that any column in the right-hand table of a LEFT JOIN can be NULL even if the column has a NOT NULL constraint. Do not apply optimizations that assume the column is never NULL.
* In PRAGMA journal_mode=TRUNCATE mode, call fsync() immediately after truncating the journal file to ensure that the transaction is durable across a power loss.
* Fix an assertion fault that can occur when updating the NULL value of a field at the end of a table that was added using ALTER TABLE ADD COLUMN.
* Do not attempt to use the strchrnul() function from the standard C library unless the HAVE_STRCHRNULL compile-time option is set.
* Fix a couple of problems associated with running an UPDATE or DELETE on a VIEW with a rowid in the WHERE clause.
Performance Enhancements:
* Many micro-optimizations result in 20.3% more work for the same number of CPU cycles relative to the previous release. The cumulative performance increase since version 3.8.0 is 61%. (Measured using cachegrind on the speedtest1.c workload on Ubuntu 13.10 x64 with gcc 4.8.1 and -Os. Your performance may vary.)
* The sorter can use auxiliary helper threads to increase real-time response. This feature is off by default and may be enabled using the PRAGMA threads command or the SQLITE_DEFAULT_WORKER_THREADS compile-time option.
* Enhance the skip-scan optimization so that it is able to skip index terms that occur in the middle of the index, not just as the left-hand side of the index.
* Improved optimization of CAST operators.
* Various improvements in how the query planner uses sqlite_stat4 information to estimate plan costs.
New Features:
* Added new interfaces with 64-bit length parameters: sqlite3_malloc64(), sqlite3_realloc64(), sqlite3_bind_blob64(), sqlite3_result_blob64(), sqlite3_bind_text64(), and sqlite3_result_text64().
* Added the new interface sqlite3_msize() that returns the size of a memory allocation obtained from sqlite3_malloc64() and its variants.
* Added the SQLITE_LIMIT_WORKER_THREADS option to sqlite3_limit() and PRAGMA threads command for configuring the number of available worker threads.
* The spellfix1 extension allows the application to optionally specify the rowid for each INSERT.
* Added the User Authentication extension.
Bug Fixes:
* Fix a bug in the partial index implementation that might result in an incorrect answer if a partial index is used in a subquery or in a view.
* Fix a query planner bug that might cause a table to be scanned in the wrong direction (thus reversing the order of output) when a DESC index is used to implement the ORDER BY clause on a query that has an identical GROUP BY clause.
* Fix a bug in sqlite3_trace() that was causing it to sometimes fail to print an SQL statement if that statement needed to be re-prepared.
* Fix a faulty assert() statement.
