- Fix for system tables in CegoAttrDesc::evalReferences, for system
tables the evalTableReferences method has to be called, otherwise
attributes for system tables can not be accessed
- Add dbCheck for Fetch & Cache ( check047 ), all base checks are
run now with querycache ON and OFF
- Fix in CegoSelect::nextTuple, the cache schema was not set up
properly, if nextTuple was just called with empty jfl. This happens
in case of procedure cursors, ( CegoProcFetch ), where the field
value list is for all method calls empty
- Fix for CegoExpr::getAggregationList(), missing break operations
in switch block. This bug might lead to error messages regarding
aggregation queries like
CGCLT > select sum(a) + 0 from t1;
Error : Invalid expression sum(t1.a) + 0 for aggregation
- Modification for admin command set tsroot. Also the base path for
all defined datafiles is set now to new tsroot
- Fix in CegoBTreeCursor and CegoBTreeManager. Several conditons
have been changed from fileId && pageId to fileId || pageId
This bug may results in invalid btree objects in case of btree
spawn over several datafiles ( the pageId might be 0 in the next
file )
- Fix in CegoGroupSpace::insertTuple, the grouping tuple has to be
searched using getTableAlias instead of getTableName, otherwise the
following query returns wrong results
select count(*), a from t1 group by a; --works
select count(*), a from t1 tx group by a; -- broken
A check for this has been added in dbcheck suite
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Introduced BTree Cache to speed up the creating of large btree
indizes. A BTree Cache test done with 100 million tuples went
from 20h to 2.6h to create the index.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
databases/cego: 2.26.14 -> 2.26.16
lfcbase:
- Improvement for ListT template class. The end of the list is now
stored int _listEnd variable. This improves Insert performance,
since the end of the list can immediately accesed instead of tracking
through the whole list until the end is reached.
- Patch in ThreadLock, the mutex for lock statistics has been
removed, since there is no need for synchronization. Also errors
have not been checked for the pthread_rwlock_wrlock, which might
cause some deadlocks.
cego:
- Ugly bug fixed in CegoObjectManager::releaseDataPtrUnlocked.
The logical operator && must be replaced by ||, otherwise fix leak arises
- Fix in CegoSelect::nextTuple for query cache to avoid, that queries
including system tables are cached.
- Error message fixes in CegoXPorter
- Introduced bufferpool distribution rate information in show pool output.
This indicates, if the buffer pool is used in a balanced way.
The behaviour can be changed with the BUFFERPOOLHASHKEY parameter.
The list pool command now receives the pool information in several
chunk messages. This allows to print also very large buffer pool
configurations
- Introduced BUFFERPOOLHASHKEY as a database tuning parameter. Based
on this parameter, the corresponding bufferpool slot for a fileId,
pageId tuple is calculated. Depending on the configured datafile
size, it might be useful to adjust this parameter
- Cego Tera Test started. Approach is to load about 1TB of random
data through the whole stack ( via cgblow ). Actually we are running
with 19 parallel processes loading concurrently 4 different tables.
Estimated date to achieve TB-Size is in about one month.
- Changed command line argument --nolockstat to --lockstat. So the
default value is lockstat disabled, since this has impact to the
database performance
- Modification of hash function in CegoBufferPool to achieve a
better distribution for large buffer pool configurations
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
devel/lfcxml: 1.1.7 -> 1.1.8
databases/cego: 2.26.13 -> 2.26.14
lfcbase:
- Added kill method to class Process, added SigTest check program to
basecheck suite
- Fix in BigDecimal::operator < and > .. comparison was not done correctly.
- Added automake check suite in baseecheck directory ( derived from samples )
lfcxml:
- Added automake check suite to package
- Small fix for XMLSuite constructor for pInStream initialization
cego:
- Added encoding implementation for CegoCaseCond to allow case
conditions also for update operations like
update erp_disk
set location = case when substr(wwn,20,1) = '1' then 'MD' else 'GV' end;
- Cosmetic changes in CegoDefs, increased FILEMNG_MAXDATAFILE to 5000
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoAttrDesc::evalReferences. In case of join objects, the
refCount was not calculated correcty, since just the join object
was analysed but not the sub content object. For this, a new method
was introduced ( evalTableReferences ) and in case of join object,
evalReferences is called recursive for all sub content objects
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- cgsimload with 3 mio ops and 2 processes passed
- Fix in CegoBTreeCursor::getFirst, if pAttrCond == 0 and btree is empty,
the method call tries to claim a zero datapointer
- Reorganization of CegoDistManager and CegoTabelManager, all
methods, which are calling useObject/unuseObject are located in
CegoTableManager class now.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Added more sophisticated corruption check for btrees
( CegoBTreeManager::verfiyBTree ), which also checks for any structure
corructions in all btree nodes
- Added btree check to CegoDistManager::verifyTable
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Version passed concurrent load test with 10 million concurrent
operations ( random insert, update and delete on one table with 2
processes )
- Fix in CegoFileHandler::allocatePage, In case of page exceed with
appendMode = true, a second try is done with appendMode = false
- Small fix in CegoObjectManager::reorgObject, ref page id information
for append mode is set to new last page id
- Small fix in CegoSelect::filterRefs, allow expressions with constant
values ( getAttrRefList= 0 ) in aggregation
- Locking improvements for delete and update operations.
All delete operations are done now via transactions ( introduced
CegoTableManager::deleteDataTableAtomic ). This allows to use a
shared lock as the table lock for using the table
(CegoDistManager::deleteLocalDataTable and
CegoDistManager::updateLocalDataTable )
As a result, a table can be accessed in parallel during a ( long
running ) delete operation on the table( e.g. delete from table).
E.g. select count(*) from table will not block anymore, rather
indicates the remaining number of tuples in the table.
This modification is still under investigation and must verfied
( also in combination with btree/index objects on the table )
- Fix in CegoBTreeManager::deleteBTree, the relevance value was not
calculated correctly
Fix in CegoQuery, cache invalidation is done before query operations.
Otherwise, if the query is aborted, an exception is thrown and
the cache is not cleaned
- Small modifications on cgblow load simulation client
- Changed locking for delete an update operations to exclusive due
to concurrency prpblems with btree operations
- Added page locking to AVLIndexManager and AVLIndexCursor
- Improvements for locking done. Since locking is done on page level,
Insert and delete operations can be done now in parallel . For this
locking had to be introduced in CegoBtreeManager::insertBTree and
CegoBTreeManager::deleteBTree
Further performance path for grouping queries in CegoGroupSpace.
Since the current grouping key is stored now in the class variable
_groupSchema, some expensive list copy operations could be avoided.
This improves performance for about 4%.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Improvements for query execution plan retrieval
- Fix in CegoOutput for multline format
( e.g. execution plans have been cut off )
- Performance patch in CegoTableCursor::setup. The getObjectListByTable
method was called every time, when the attribute condition has been
changed. Since this operation is pretty expensive, it was useful
to cache the retrieved index/btree objet in a local class variables.
This change results in significant query speedup for joins, where
the table cursor has to be setup very often
- Small fix in CegoAction::readChain, treating backslashes for
quoteEscapeFlag=false was commented, reactivated
- Small fix in CegoMain, default for csmode changed to value "ID"
- Fix for CegoBtreeManager::insertBTree. Duplicate handling did not
work correctly, if CegoBTreeNode::addValue returns false ( node is
full ) with a duplicate detection, we also have throw exception.
Otherwise the node is split and a duplicate is added.
- Performance improvement for nested queries ( exists select,
in select). The tablecursor for the select statement is not
released and reallocated for subsequent queries ( in CegoQueryHelper,
the select handle is reset with argument false ) This has required
a different locking strategy for the cursor object in CegoDistCursor.
The cursor object has explicit to get in use, if the nextTuple
method is called the first time.
- Added online help for query cache and table cache in cgadm
- Added further performance fix for aggregation queries in CegoExpr
and CegoQueryHelper. The aggregation list for CegoExpr now is stored
in a class variable and is just evaluated the first time, it is
retrieved. In the method CegoQueryHelper::aggregateTuple, the
increment calculation of the count aggregation has been changed
from memcpy calls to direkt pointer arithmetic
- Performance fix has been done for record locking. In many cases,
row locking could be avoided, since an appropriate locking is ensured
on data page or object level. This brings another significant
performance improvement.
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoBufferPool::calcHas and CegoBufferPool::calcSegement
For large file id's the integer range for the hash key might be
exceeded, which lead to wrong address calculation.
The hash key now is calculated in long based values
- Introduced server mode with same behaviour as daemon mode, but
no child process is forked and server can be terminated with Ctrl-C
- Small formatting fix in CegoOutput
- Fix in CegoDistCursor for inner and outer joins
Inner or outer joins with additional where condition might return
incorrect result set since the condition was just evaluated inside
the join.
To correct this, the WHERE condition is also checked via evalCondition.
This required also a fix in the evalCondition method.
- Added multi segment support to CegoBufferPool. Since the pool is
divided now into several memory segments, this allows larger buffer
pool configurations.
- Improvements for output formatting regarding string functions
(cegoFunction::getReturnTypeLen). Size of required column field size
is now calculated, so the outlook looks more pretty
- Added hints from David Binderman in CegoAction::backChar and
CegoBufferPage::printPage ( changed pointer comparison to avoid
-Wextra compile warnings ) Thanks !
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Decreased default cache size values to 100k ( query + table )
- Next fileId (CegoXMLSpace::nextFID ) calculation now analyses the
existing fileId and uses the lowest available. The MAXFID attribute
in the database xml file is no more used. This avoids, that file
id exceeds in case of frequent creation and removement of tablesets.
- Some format correction in CegoAdminHandler
- Increased parameter FILMNG_MAXDATAFILE to 10000 and
TABMNG_MAXTABSET to 500
- Optimization added in CegoDisCursor::distSetup(CegoAttrCond)
If the native table cursor could not set up either FULL or PART, the
table cursor retrieved all available rows from the corresponding table.
In this case, the table also can be cached to improve subsequent
queries for this table. This fix is relevant for queries like
select * from t1 where b like '%X%', since the like condition
cannot be used by the table cursor
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoSelect::nextTuple : in case of rowlimit > 0,
the cached entry was not added to query cache
- Added many columns sample to base test
( create, insert and select for table with 60 columns )
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- Fix in CegoAction::execViewCreate.
Views must be compiled successful now immediately after creation.
Otherwise, the calculated schema is not stored correctly into
dictionary.
- Fix in CegoFactor::getId.
If sysdate is used in queries, it should be resolved, otherwise
query couldbe used from cache which returns invalid result
Submitted by: Bjoern Lemke <lemke@lemke-it.com>
- In CegoQueryHelper::evalPredicate fixed termination for INSUB- and
NOTINB queries. There was no reset at the end of the evaluation, which
might lead to open table cursors. In case of procedure queries, this
might lead to locking problems.
Since the query is reset after evaluation, query caching is no more
useful. This must be treated in an advanced way.
- Further patches in CegoTableManager::deleteDataTable,
CegoFactor::evalReferences and CegoSelect::nextTuple for advanced
query cache handling. All patches have impact on queries of the form
DELETE FROM t1 WHERE a IN ( SELECT ....)
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Added performance patch in CegoFieldValue.
Values with a length up to STATICFIELDBUF size ( 100 bytes ) are handled
using a static byte array. This saves expensive memory allocation
calls in most cases. First very rough performances analysis has
shown a performance improvement of about 5-10 percent
- The local copy logic was shifted from CegoSelect::nextGroupedTuple
to CegoGroupSpace::insertTuple.
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Fix in CegoSelect::nextGroupedTuple.
For the 'grouping key' values, a local copy of CegoFieldValue has to be
created. Otherwise, grouping references might be invalidated, which result
in aborted grouping queries with strange error message ( No token
available ) on client side.
For local copy creation, the CegoFieldValue::getLocalCopy method has
been modified ( was not used until now ).
This effect was observed with small buffer poolsizes in
combination with grouping queries on large tables. The probability
of grouping key invalidation is higher in that case, because
buffer pool pages are reloaded more often.
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Performance fix in CegoSelect::consolidateConjunctionList
Inappropriate predicates are filtered and added to stack first.
As a result, predicates are used for table join binding,
so that efficient tuple retrievel is ensured.
- The fix also avoids infinite query loops for non-reference conditions,
e.g. select a from t1 where 1 = 1;
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Simplification in CegoBTreeManager and CegoBTreeNode.
Calculation of leaf node chain is now much simpler. This ensures
consistency and improves btree build up performance.
- Fix for btree page count and btree page release. Some of the btree
nodes have double references, which was not treated in the page count
and page release algorithm.
- Changed default primary and secondary values for tableset definition
command to the configured hostname. This simplifies the define command
in case of single node tableset configurations.
Submitted by: Björn Lemke <lemke@lemke-it.com>
databases/cego: 2.23.6 -> 2.23.7
lfcbase: Fix in Chain::visibleLength.
Calculation may result in an infiite loop, if mbtowc returns zero.
cego:
- Fix in CegoFieldValue for VARCHAR comparison. In case of exceeded
string length for index values, the CegoFieldValue did not work
correctly.
- Fix in CegoClient for dump mode. BTrees habe not been dumped with
the correct type ( primary, unique, normal )
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Fix in CegoSelect::buildJoinTables.
After growStep match, growStep is now reset to 1.
This avoids to add tables to the joinlist, which are still not
treated for any appropriate join condition. As an effect of this
bug, execution plan is not evaluated in an optimal manner, which
might lead to long execution times
- Fix in CegoTableManager::insertDataTable, in case of constraint
violation, btrees have not been cleaned up. This may result in
inconsistent btree objects.
- Added a fix in CegoTableManager::updateTuple.
The core operation consists of a deleteDataTable and a
insertDataTable method call. For the insertDataTable method call,
the logging still was enabled. So in addition to an update
operation, also an insert operation was done. In case of recovery
operations, this lead to duplicate entries. Fix:
just change isLogging parameter value for insertDataTable method
from true to false.
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Added uptime command for cgclt to show database uptime
- Added pool uptime to buffer pool information, this indicates the
database uptime
- Several fixes for cgmkdb utility
Submitted by: lemke@lemke-it.com
- Introduced CegoDatabaseFormater.
Since the API has changed (proagation of database formatting methods),
this is 2.23.0
Submitted by: Björn Lemke <lemke@lemke-it.com>
databases/cego: 2.21.1 -> 2.22.0
databases/p5-DBD-cego: 1.2.9 -> 1.3.0
lfcbase:
- Added syslog support
- syslog is enabled per default
cego:
- fix in CegoFieldValue::comp for correct null value ordering
- adapt to lfcbase 1.6 with syslog support
p5-DBD-cego:
- adapt to lfcbase 1.6 with syslog support
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Fixes for btree handling. Some field value offsets have been corrected
based on experiences with SAGE db import
- Improved error messages for CegoLockHandler to indicate low level lock
error messages from pthread calls
- Added sudo command to cego rc script to run db as given user
Submitted by: lemke@lemke-it.com
- Urgent fix in CegoFieldValue::comp method.
New introduced null value comparison might result in a dump.
Since, the new btree null value extension breaks specification
to version 2.20, update to version 2.21 was required.
Submitted by: lemke@lemke-it.com
- Added multibyte character evaluation in CegoOutput to show visible
chain length
- added build_depends for a specific version of lfcbase
Submitted by: lemke@lemke-it.com
- Fix on update query for referenced attributes in nested select like
update t1 tx set b = ( select d from t2 where t1.a = c )
This table alias was not resolved correctly.
Submitted by: lemke@lemke-it.com
- SQL optimizer improvement was added. In case of select conditions
with "a is null", neither index nor btree was used. Just for "a = null",
the index was used. This has been fixed.
- Added fix for btree null value handling. A dedicated null value
flag was introduced for each btree value ( class CegoBTreeValue )
to handle null values.
Submitted by: lemke@lemke-it.com
- Added command line parameter validation for poolsize, numdbthread,
numadminthread and numlogthread parameters
- Fix in CegoFactor::evalFieldValue : For the QUERY case, the select
instance was not reset. Since the query starts from a undefined point,
this might lead to "list position exceeded" exceptions
- Fix in CegoQueryHelper::evalSelection for isAgg-case
Submitted by: Björn Lemke <lemke@lemke-it.com> (author)
Change for primary and secondary host evaluation.
Instead of comparing gethostbyname, the defined dbhost value is used.
This helps if the configured dbhost does not match with the result
value of gethostbyname (e.g. is multiple hostnames are configured
on a database server)
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Added restricted access for JDBC information requests
The jdbc role now is required to perform any JDBC information requests.
- A problem occured with inner and outer joins if the same table
occurs two times or more in the query
For the CegoAttrComp class, the tablealias name is stored now to
evaluate if the compare is appropriate for the corresponding table
- Small optimization in CegoDisManager::updateLocalDataTable
Instead of requesting a new lock for the commitment phase, the lock we
already have is also used for the commit in case of forecedTransactions
( update case ).
- This ensures, that the forced transaction can be done without
risking to not obtain the lock for the corresponding table again.
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Fix in CegoQueryHelper::decodeFVL to treat also decodings for
field value list in switched order ( btree or index fvl )
- Fix in CegoAction::execIndexCheck to conform to new object comparison
in CegoDatabaseManager::ObjectRecord
- Fix in CegoTableManager::deleteDataTable to check for inappropriate
AttrCond before executing tablecursor setup
- Fix in CegoDatabaseManager::Object::operator == Index comparisons
corrected ( index can either match uindex, pindex or index, btree
can either match ubtree, pbtree or btree )
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Fix in CegoQueryHelper::evalAttrCond. In some table ordering cases,
the attributes had been switched and the evaluation was wrong
( LESS_THAN, MORE_THAN, LESS_EQUAL_THAN, MORE_EQUAL_THAN ).
Submitted by: Björn Lemke <lemke@lemke-it.com>
- Recovery added for check constraints and foreign keys
- Corrections for btree recovery, btree creation was not performed
correctly during tableset recovery. Tableset autocorrect mode is
enabled on default now ( by define tableset )
Submitted by: Björn Lemke <lemke@lemke-it.com>
