Xapian-core 1.0.18 (2009-02-14):
API:
* Document: Add new add_boolean_term() method, which is an alias for add_term()
with wdfinc=0.
* QueryParser:
+ Add support for quoting boolean terms so they can contain arbitrary
characters (partly addresses ticket#128).
+ Add ENCLOSING_MARK and COMBINING_SPACING_MARK categories, plus several
zero-width space characters, as phrase generators. This mirrors a better
fix in 1.1.4, but without losing compatibility with existing databases.
+ Fix handling of an explicit AND before a hated term (foo AND -bar).
(ticket#447)
* TermIterator: Only include trailing '+' or '#' on a term if it isn't followed
by a word character (makes more sense and matches QueryParser's behaviour).
(ticket#446)
* Database: Fix many methods to behave better on a database with no
subdatabases, such as is constructed by Database(). Fixes ticket#415.
testsuite:
* Add test coverage for xapian-compact, and improve coverage for
WritableDatabase::replace_document().
* apitest: Rename matchfunctor<n> to matchdecider<n> to match current
terminology.
flint backend:
* When updating documents, don't update posting entries which haven't changed.
Largely fixes ticket #250.
* If the number of entries in the position table happened to be 4294967296 or
an exact multiple, Xapian would ignore positional data for that table when
running queries.
* Iterating all the terms in the database with a prefix is now slightly more
efficient.
* Fix locking code to work if stdin and/or stdout have been closed.
* If a document is replaced with itself unmodified, we no longer increase the
automatic flush counter.
* When iterating a posting list modified since the last flush(), the reported
wdf is now correct (previously it was too high by its old value).
* Replacing a document deleted since the last flush failed to update the
collection frequency and wdf, and caused an assertion failure when assertions
were enabled.
* WritableDatabase::replace_document() didn't always remove old positional
data (the only effect is that the position table was bloated by unwanted
entries).
* xapian-inspect:
+ New "until" command which shows entries until a specified key is reached.
+ New "open" command which allows easy switching between tables.
* xapian-compact: Fix typos in --help output.
quartz backend:
* Replacing a document deleted since the last flush failed to update the
collection frequency and wdf, and caused an assertion failure when assertions
were enabled.
* WritableDatabase::replace_document() didn't always remove old positional
data (the only effect is that the position table was bloated by unwanted
entries).
remote backend:
* Throw UnimplementedError if a MatchDecider is used with the remote backend.
Previously Xapian returned incorrect results in this case.
build system:
* configure: With --enable-maintainer-mode, enable -Werror for GCC >= 4.1
rather than >= 4.0 as Apple's GCC 4.0 gives bogus uninitialised variable
warnings.
documentation:
* The API documentation now includes Xapian::Error and subclasses, and doesn't
mention Xapian::Query::Internal.
* Make clear in the Xapian::Document API documentation that this class is a
lazy handle and discuss the issues this can cause.
* INSTALL: Improve text about zlib dependency.
* HACKING: Add details of our licensing policy for accepting patches.
examples:
* quest: If no database is specified, still parse the query and report
Query::get_description() to provide an easy way to check how a query parses.
portability:
* Fix GCC 4.2 warning.
xapian-core 1.0.17 (2009-11-18):
API:
* QueryParser:
+ Fix handling of a group of two or more terms which are all stopwords which
notably caused issues when default_op was OP_AND, but could probably
manifest in other cases too. Fixes ticket#406.
+ Fix interaction of FLAG_PARTIAL and FLAG_SYNONYM. (ticket#407)
* Database: A database created via the default constructor no longer causes a
segfault when the methods get_metadata() or metadata_keys_begin() are called.
flint backend:
* Don't try to close the fd one more than the maximum allowable when locking
the database. Harmless, except it causes a warning when running under
valgrind. (ticket#408)
remote backend:
* Xapian::Sorter isn't supported with the remote backend so throw
UnimplementedError rather than giving incorrect results. (ticket#384)
* Fix potential reading off the end of the MSet which is returned internally
by the remote server.
documentation:
* Various documentation comment improvements for the Database class.
examples:
* examples/quest.cc: Tighten up the type of the error we catch to detect an
unknown stemming language.
portability:
* xapian-config: Need to quote ^ for Solaris /bin/sh.
* configure: Actually use any flags we determine are needed to switch the
compiler to proper ANSI C++ mode, when building xapian-core - this stopped
working in 1.0.12, breaking support for HP's aCC, Compaq's cxx, Sun's CC, and
SGI's CC.
flint backend:
* Fix a typo which stopped this fix in 1.0.12 from working (ticket #398):
If we fail to get the lock after we spawn the child lock process (the common
case is because the database is already open for writing) then we now clean
up the child process properly.
documentation:
* Improve API documentation of QueryParser::set_default_op() and
QueryParser::get_default_op().
portability:
* Fix build failure on Mac OS X 10.6.
flint backend:
* Backport the lazy update changes from 1.1.2:
WritableDatabase::replace_document() now updates the database lazily in
simple cases - for example, if you just change a document's values and
replace it with the same docid, then the terms and document data aren't
needlessly rewritten. Caveats: currently we only check if you've looked at
the values/terms/data, not if they've actually been modified, and only keep
track of the last document read.
* Fix PostingIterator::skip_to() on an unflushed WritableDatabase to skip
documents which were added and deleted since the last flush. (ticket#392)
documentation:
* Overhaul the doxygen options we use and tweak various documentation comments
to improve the generated API documentation.
* Explicitly document that an empty prefix argument to
QueryParser::add_prefix() means "no prefix".
* Update the documentation comments for Enable::set_sort_by_value(),
set_sort_by_value_then_relevance(), and set_sort_by_relevance_then_value() to
mention sortable_serialise() as a good way to store numeric values for
sorting.
API:
* When using more than one ValueRangeProcessor, QueryParser didn't reset the
begin and end strings to ignore any changes made by a ValueRangeProcessor
which returned false, so further ValueRangeProcessors would see any changes
it had made. This is now fixed, and test coverage improved.
flint backend:
* Use F_FULLFSYNC where available (Mac OS X currently) to ensure that changes
have been committed to disk. (ticket#288)
remote backend:
* Fix handling of percentage weights in various cases when we're searching
multiple remote databases or a mix of local and remote databases.
API:
* Xapian::Document no longer ever stores empty values explicitly. This
wasn't intentional behaviour, and how this case was handled wasn't
documented. The amended behaviour is consistent with how user metadata
is handled. This change isn't observable using Document::get_value(),
but can be noticed when iterating with Document::values_begin(), using
Document::values_count(), or trying to delete the value with
Document::remove_value().
matcher:
* If a query contains a MatchAll subquery, check for it before checking the
other terms so that the loop which checks how many terms match can exit
early if they all match.
* When an OR or ANY_MAYBE decayed to an AND, we were carefully swapping the
children for maximum efficiency, but the condition was reversed so we were
in fact making things worse. This was noticed because it was resulting in
the same query running faster when more results were asked for!
* Only build the termname to termfreq and weight map for the first subdatabase
instead of rebuilding it for each one. Also don't copy this map to return
it. This should speed up searches a little, especially those over multiple
databases.
* If a submatcher fails but ErrorHandler tells us to continue without it, we
just use a NULL pointer to stand in rather than allocating a special dummy
place-holder object.
* Remove AndPostList, in favour of MultiAndPostList. AndPostList was only used
as a decay product (by AndMaybePostList and OrPostList), and doesn't appear
to be any faster. Removing it reduces CPU cache pressure, and is less code
to maintain.
* Call check() instead of skip_to() on the optional branch of AND_MAYBE.
flint backend:
* Fix a bug in TermIterator::skip_to() over metadata keys.
remote backend:
* Fix xapian-tcpsrv --interface option to work on MacOS X (ticket#373).
* Fix typo which caused us to return the docid instead of the maximum weight
a document from a remote match could return! This could have led to wrong
results when searching multiple databases with the remote backend, but
probably usually didn't matter as with BM25 the weights are generally small
(often all < 1) while docids are inevitably >= 1.
inmemory backend:
* The inmemory backend doesn't support iterating over metadata keys. Trying
to do so used to give an empty iteration, but has now been fixed to throw
UnimplementedError (and this limitation has now been documented).
documentation:
* INSTALL: We no longer regularly test build with GCC 2.95.4 and we're raising
the minimum GCC version required to 3.1 for Xapian 1.1.x.
* Document what passing maxitems=0 to Enquire::get_mset() does.
* docs/queryparser.html: Add examples of using a prefix on a phrase or
subexpression.
* Correct doxygen comments for user metadata functions:
Database::get_metadata() throw UnimplementedError but
WritableDatabase::set_metadata() can.
* Document that Database::metadata_keys_begin() returns an end iterator if the
backend doesn't support metadata.
* HACKING: Update the list of Debian/Ubuntu packages needed for a development
environment.
* WritableDatabase::remove_spelling() now works properly.
* The QueryParser now treats NON_SPACING_MARK Unicode characters as phrase
generators, which improves handling of Arabic. This is a stop-gap solution
for 1.0.x which will work with existing databases without requiring
reindexing - in 1.1.0, NON_SPACING_MARK will be regarded as part of a word.
(ticket#355)
* Fix undefined behaviour in distribution of OP_NEAR and OP_PHRASE over a
non-leaf subquery (indentified by valgrind on testcase nearsubqueries1).
(ticket#349)
* Enhance distribution of OP_NEAR/OP_PHRASE over non-leaf subqueries to work
when there are multiple non-leaf subqueries (ticket#201).
* Enquire::get_mset() no longer needlessly checks if the documents exist.
* PostingIterator::get_description() output improved visually in some cases.
* Enquire::get_mset():
+ Now throws UnimplementedError if there's a percentage cutoff and sorting is
primarily by value - this has never been correctly supported and it's
better to warn people than give incorrect results.
+ No longer needlessly copies the results internally.
+ When searching multiple databases, now recalculates the maximum attainable
weight after each database which may allow it to terminate earlier.
(ticket#336).
+ Fix inconsistent percentage scores when sorting primarily by value, except
when a MatchDecider is also being used; document this remaining problem
case. (ticket#216)
* Enquire::set_sort_by_value() (and similar methods): Rename the wrongly named
"ascending" parameter to "reverse", and note that its value should always be
explicitly given since defaulting to "reverse=true" is confusing and the
default will be deprecated in 1.1.0. (ticket#311)
* Database::allterms_begin(): Fix memory leak when iterating all terms from
more than one database.
* Query::get_terms_begin(): Don't return "" from the TermIterator (happened
when the query contained or was Query::MatchAll).
* Add QueryParser::FLAG_DEFAULT to make it easier to add flags to those set by
default.
This changes the buildlink3.mk files to use an include guard for the
recursive include. The use of BUILDLINK_DEPTH, BUILDLINK_DEPENDS,
BUILDLINK_PACKAGES and BUILDLINK_ORDER is handled by a single new
variable BUILDLINK_TREE. Each buildlink3.mk file adds a pair of
enter/exit marker, which can be used to reconstruct the tree and
to determine first level includes. Avoiding := for large variables
(BUILDLINK_ORDER) speeds up parse time as += has linear complexity.
The include guard reduces system time by avoiding reading files over and
over again. For complex packages this reduces both %user and %sys time to
half of the former time.
rationalisation of backends (and also allows wiring the database to the older
database format if desired).
The "suggested options" select support for both formats, which is also the
status quo.
Xapian-core 1.0.10 (2008-12-23):
API:
* Composing an OP_NEAR query with two non-term subqueries now throws
UnimplementedError instead of AssertionError (in a --enable-assertions build)
or leading to unexpected results (otherwise). This partly addresses bug#201.
* Using a MultiValueSorter with no values set no longer causes a hang or
segmentation fault (but it is still rather pointless!)
matcher:
* If we're using values for sorting and for another purpose, cache the
Document::Internal object created to get the value for sorting, like we do
between other uses.
flint backend:
* If the disk became full while flushing database changes to disk, the
WritableDatabase object would throw a DatabaseError exception but be left in
an inconsistent state such that further use could lead to the database on
disk ending up in a "corrupt" state (theoretically fixable, but no tool
to fix such a database exists). Now we try to ensure that the object is
left in a consistent state, but if doing so throws a further exception, we
put the WritableDatabase object in a "closed" state such that further
attempts to use it throw an exception.
* Create the lockfile "flintlock" with permissions 0666 so that the umask is
honoured just like we do for the other files (previously we used 0600).
Previously it wasn't possible to lock a database for update if it was
owned by another user, even if you otherwise had sufficient permissions via
"group" or "other".
* Fix garbled exception message when a base file can't be reread.
quartz backend:
* Fix garbled exception message when a base file can't be reread.
remote backend:
* xapian-tcpsrv and xapian-progsrv now accept -w as a short form of --writable,
as was always intended.
build system:
* This release now uses newer versions of the autotools (autoconf 2.62 ->
2.63; automake 1.10.1 -> 1.10.2).
documentation:
* INSTALL: Add new paragraphs about HP's aCC and IRIX (adapted from footnotes
in PLATFORMS).
* PLATFORMS: HP testdrive has been shut down, so all mark all those machines as
"no longer available". Update atreus' build report to 1.0.10.
* docs/queryparser.html: Add link to valueranges.html.
examples:
* delve: Add missing "and" to --help output. Report termfreq and collection
freq for each term we're asked about.
portability:
* Fix to build with GCC 4.4 snapshot.
Xapian-core 1.0.9 (2008-10-31):
API:
* Database::get_spelling_suggestion() is now faster (15% speed up for parsing
queries with FLAG_SPELLING_CORRECTION set in a test on real world data).
* Fix OP_ELITE_SET segmentation fault due to excess floating point precision
on x86 Linux (and possibly other platforms).
* Database::allterms_begin() over multiple databases now gives a TermIterator
with operations O(log(n)) rather than potentially O(n) in the number of
databases.
* Add new Database methods metadata_keys_begin() and metadata_keys_end() to
allow the complete list of metadata in a database to be retrieved (this
API addition is needed so that copydatabase can copy database metadata).
testsuite:
* Remove the cached test databases before running the testsuite.
* apitest: Fix cursordelbug1 to work on Microsoft Windows (bug#301).
* apitest,queryparsertest: Skip tests which fail because the timer granularity
is too coarse to measure how long the test took. In practice, this is only
an issue on Microsoft Windows (bug#300 and bug#308).
matcher:
* Adjust percent cutoff calculations in the matcher in a way which corresponds
to the change to percentage calculations made in 1.0.7 to allow for excess
precision.
* Query::MatchAll no longer gives match results ranked by increasing document
length.
flint backend:
* xapian-compact: Fix crash while compacting spelling table for a single
database when built with MSVC, and probably other platforms, though Linux
got lucky and happened to work (bug#305).
build system:
* configure: Disable -Wconversion for now - it's not useful for older GCC and
is buggy in GCC 4.3.
* configure: Set -Wstrict-overflow to 1 instead of 5, to avoid unreasonable
warnings under GCC 4.3.
documentation:
* Minor improvements to API documentation, including documenting the
XAPIAN_FLUSH_THRESHOLD environmental variable in WriteableDatabase::flush()
(bug#306).
* valueranges.html: Fix typos in example code, and drop superfluous empty
destructor from ValueRangeProcessor subclass.
* HACKING: Several improvements.
examples:
* copydatabase: Also copy user metadata.
Xapian-core 1.0.8 (2008-09-04):
API:
* Fix output of RSet::get_description
testsuite:
* Report subtotals per backend, rather than per testgroup per backend to make
the output easier to read.
flint backend:
* Fix WritableDatabase::add_document() and replace_document() not to be O(n*n)
in the number of values in the new document.
* Fix handling of a table created lazily after the database has had commits,
and which is then cursored while still in sequential mode.
* Fix failure to remove all the Btree entries in some cases when all the
postings for a term are removed. (bug#287)
* xapian-inspect: Show the help message on start-up. Correct the documented
alias for next from ' ' to ''. Avoid reading outside of input string when it
is empty. (bug#286)
quartz backend:
* Backport fix from flint for WritableDatabase::add_document() and
replace_document() not to be O(n*n) in the number of values in the new
document.
build system:
* configure: Report bug report URL in --help output.
* xapian-config: Report bug report URL in --help output.
* configure: Fix deprecation error for --enable-debug=full to say to instead
use '--enable-assertions --enable-log' not '--enable-debug --enable-log'.
documentation:
* valueranges.html: Expand on some sections.
examples:
* quest: Fix to catch QueryParserError instead of const char * which
QueryParser threw in Xapian < 1.0.0.
* copydatabase: Use C++ forms of C headers. Only treat '\' as a directory
separator on platforms where it is. Update counter every 13 counting up to
the end so that the digits all "rotate" and the counter ends up on the exact
total.
portability:
* Eliminate literal top-bit-set characters in testsuite source code.
released under the GPL. It's written in C++, with bindings to allow
use from Perl, Python, PHP, Java, Tcl, C# and Ruby (so far!)
Xapian is a highly adaptable toolkit which allows developers to
easily add advanced indexing and search facilities to their own
applications. It supports the Probabilistic Information Retrieval
model and also supports a rich set of boolean query operators.
If you're after a packaged search engine for your website, you
should take a look at Omega: an application we supply built upon
Xapian. Unlike most other website search solutions, Xapian's
versatility allows you to extend Omega to meet your needs as they
grow.
