With Google Calendar, having a VALARM leads to different communication
compared to the case without VALARM (requires resending). Test both
cases by providing multiple sets of linked items.
Client::Source now contains LinkedItems_1 and LinkedItems_2
sub-groups. This required changes in resultchecker.py, to keep these
tests as listed in the same table as the other source tests.
ClientTestConfig becomes more complicated: it used to be a plain C
struct which could be copied/cleaned with memcpy/memset. This approach
is kept by adding a pointer to a std::vectore. A nicer solution would
be to turn all "const char *" into std::string and int/bool values
with wrapper classes which initialize them.
Instead of aborting the test at the first item which
fails to import, collect the error messages (from the
exceptions), do the comparison of imported data
against the reference data and then either report
the import failures (if there were any) or the comparison
failure (if that broke).
The advantage is that a single run of the test shows
all problems that exist.
Client::Source::*::testLinkedItems* tested local storage operations
for events sharing the same UID. For syncing, only testItems had
a little bit like it. This commit adds more tests specifically
designed to exercise corner cases of syncing such linked items:
- sync parent, then sync with child and other event added on different
sides (corner case in CalDAV change tracking, BMC #22329)
- sync child
- sync child, sync parent later
Trigger "item already deleted" in syncing, check that syncing
completes and leaves no items around.
Somehow the problem in BMC #22327 must have been different: when
running the test, the Synthesis engine recognizes that the item
was removed on both sides and logs:
So this test doesn't really cover BMC 22327. But it is worthwile
to run it anyway.
The distinction between vcard21 and vcard30 became mute in the Evolution
backend a while ago. Both tests ended up using the vCard 3.0 Evolution
tests data and the default uri for each server. This patch removes
the vCard 2.1 special case.
It also renames the tests and test data to reflect that they always
were Evolution specific. The new naming convention, also applied
to file, QtContacts, KCalExtended, XMLRPC, Maemo and Akonadi backends, is
now <backend>_contact/event/task/memo, with eds/file/qt/kcal/maemo/kde
as backend names.
The reasoning is:
- results in unique string (in particular no overlap with
backend type names), easier to search for
- underscore already used before (in contrast to hyphen)
- no plural-s to keep the name shorter
The Akonadi backend should be using its own test data instead of
the Evolution ones.
Creating test data in SyncTests::doVarSizes() took a long time
because it created a TestingSyncSource multiple times inside
a loop. Each destruction of that instance caused a 5 second delay,
necessary to ensure correct time-based change tracking.
Adding a second version of insertManyItems() which uses a single
instance of TestingSyncSource created by doVarSizes() itself
makes it possible to sleep only once.
One of the features of the Synthesis engine is that it can preserve
properties locally which are not supported by the
peer. "testExtensions" covers that, by updating all items on the
server (via client B) and reimporting them as updates into client A.
The data comparison then is done without the usual "allow server to
drop data" simplifications in synccompare.
This test is not enabled yet. "config.update" needs to be set for it,
but doing so then trips over other changes introduced by servers, like
for example re-encoding photos. Needs some more thoughts and testing...
This test verifies that timeout detection works for SyncML/HTTP and
local sync. The test does not depend on the source which is active,
so to cut down redundant testing it registers itself only for
the first source in CLIENT_TEST_SOURCES.
The implementation of the test creates a dead listening socket which
the HTTP transports can connect to without ever getting a reply. Then
the syncURL for outgoing HTTP is redirected towards that dead port.
The expectation is that the overall sync times out roughly after 10
seconds.
Instead of allowing users of the API to register a callback
which can choose between aborting and time out, only accept
the timeout duration and always treat that as a timeout.
The advanced functionality wasn't used and the simplification
makes implementing the API easier.
Compared to the existing Resend tests, the new ResendProxy tests have
the advantage that they cover one additional error case: loss of
connection before the server has a chance to send the full reply. This
happened to be one case that the syncevo-http-server wasn't handling
properly.
This failure cannot be simulated inside client-test because the
TransportAgent API doesn't provide enough control. The implementation
uses a twisted HTTP proxy which intercepts data transfer at three
different points (before sending, after sending and before receiving,
after receiving) and drops the client connection. The script listens
on four different ports, numbered consecutively, with the first one
not introducing errors, and all others representing one failure.
The test must set and unset proxy information in the underlying transport;
this patch makes that possible in SoupTransportAgent::setProxy().
To use the new tests, run "proxy.py <port>", then set
CLIENT_TEST_RESEND_PROXY=<port>.
When a sync completes, CLIENT_TEST_LOG=<path> copies everything it
finds at that path into "server-log" inside the sync's log
directory. This is useful when testing against a local server with its
own logs.
To use it with SyncEvolution, configure logdir=/tmp/server-log in the
server's configurations for the test clients and then set
CLIENT_TEST_LOG=/tmp/server-log.
To enable it in configuration, use '--enable-buteo-tests'.
To test client-test with buteo to do sync, make sure the
environment variable 'CLIENT_TEST_BUTEO=1' is set.
Three main steps to support buteo testing in client-test,
1) pre-run: storage preparation, clean up buteo logging
files and set corresponding keys in meego-sync-conf.xml.
For qtcontacts, switch backend database files for each
client; For calendar/todo/note, specify different notebook
names for each client.
2) run: re-implement ClientTest.doSync function, which
dbus-calls 'msyncd', the dbus daemon of buteo to do sync.
Listen to needed signals to track its status.
3) post-run: collect its status and summarize its statistics
and convert to internal sync report.
Buteo doesn't support explicit slow-sync mode and refresh-
from-server mode. Add two workarounds for them:
For slow-sync mode, wipe out anchors;
For refresh-from-server, wipe out anchors and clean all local
data and do two-way sync.
To enable buteo testing in your environment, you have to:
1) Add read/write privilege for the directory /etc/sync for
current testing user
2) To test one remote server, make sure the profile xml is put
in $HOME/.sync/profiles/sync. The name of the profile xml should
be set as the server name.
This patch enhances the tests by also comparing the actual data on the
server against the expected content. This was made possible by returning
the generated test data back to the test function, which then uses
the new compareDatabases() utility function to do the comparison.
Previously, test data was modified once when creating the test
configuration. That turned out to be too limited for Google
Calendar, which requires operations like "bump SEQUENCE number"
each time a test case is used.
This patch therefore introduces a "mangleItem" callback. In most
cases, it doesn't do anything. But for iCalendar 2.0, it applies the
transformations previously found in the config setup (simplify UID,
...) and adds some new ones (increase SEQUENCE number, set
LAST-MODIFIED to current time, avoid conflicts between different test
runs by making the UID unique to the run if CLIENT_TEST_UNIQUE_UID is
set).
The patch also fixes time stamps in the test data: they should have been
in UTC, but lacked the Z suffix.
Destroying ORBit based Evolution backends at the very end of the
process life time caused assertions in ORBit. Added explicit cleanup
code which is called before triggering the normal library shutdown by
leaving main().
The suspend tests would have passed even if the client never really
executed the suspend request. Now the tests verify whether the next
session really resumes. It does that except for some corner cases
(first message, last message), which is acceptable. When multiple
sources are involved, first and last message of each source
cannot be identified reliably, so in that case suspend is not checked.
The SyncReport must be available for this checking. The check() method
was made virtual to implement such checks in a derived class, but that
is neither convenient nor does it currently work because SyncOptions
copies the CheckSyncReport instance.
The variant of the tests which transfer big items also make
sense for the suspend tests which request a suspension of the
session.
Now they are only missing from the resend tests where running
larger sessions doesn't make that much sense, at least not
for SyncEvolution and Synthesis servers.
The testInterrupt* tests simulate a hard transport failure. Each time
that happened, SyncContext would retry at least once after the default
retry period (1 minute) before giving up, slowing down the testing
considerably.
With the new SyncOptions "retryInterval" set to zero, retrying is
disabled. This must not be done when testing the resending itself.
The tests are expected to run with a maximum message size of 20000
used by both client *and* server - server must be configured
accordingly.
Then the Add and Update operations in the interrupted sync with
client B are done with an item that is larger than the message size
and thus depends on <MoreData> support. On resume, only the second
half is sent and the recipient must assemble the complete item.
testManyItems now checks that the statistics are correct. This
is based on the assumption that all active sources use the same
number of test items.
The new testManyDeletes triggers a problem with 222 loop detection:
when the message to the server contains many deletes, the server
may require so many reply messages, that the client has to send
5 Alert 222 messages in a row and incorrectly treats this as a loop
that has to be aborted.
This TransportAgent API revision was done in preparation for transport
agents which do not support HTTP. Forcing them to provide HTTP specific
methods is unnecessary, so now setting things like proxy is done when
constructing an HTTP-based agent in the createTransport() call.
So now the result of createTransport() must be ready for sending messages.
setURL() is still part of the API, because it provides message-specific
information. Perhaps it should be renamed when it is clearer what
the corresponding information in other transports is.
For connection oriented agents a new shutdown() call was introduced.
This gives them a chance to close the connection and inform the
engine about errors during that shutdown.
For servers (which send the last message without expecting a reply),
wait(noReply=true) was added.
The intention is to get rid of the historic and inconsistent
naming of some classes and their corresponding files:
* EvolutionSyncClient = class derived from Funambol's SyncClient,
* SyncEvolutionConfig = SyncEvolution's config
With the strict 'namespace SyncEvo' and the syncevo/ path prefix for
most header files it is no longer necessary to have "SyncEvolution" or
"Evolution" in the names. This patch thus renames as follows:
EvolutionSyncClient => SyncContext
EvolutionSmartPtr => SmartPtr
SyncEvolutionCmdline => Cmdline
SyncEvolutionConfig => SyncConfig
SyncEvolutionUtil => util
The former EvolutionSyncClient always had a role that went beyond just
running a sync, for example it also provided config access. With the
upcoming server support it also won't be just a client. Thus the new
name "SyncContext".
The 'syncevo/' prefix is used throughout the code now.
removed whenever the prefix made it clear that the file belongs
to SyncEvolution. This helps finding incorrect include paths.
Quotes should be used exclusively for SyncEvolution files which don't
have a specific prefix yet (test.h, config.h) to help identifying
them.
Added syncevo/declarations.h, which has
This is now used for all SyncEvolution source files, except
for the GTK UI, which is written in plain C. In the library
it helps to avoid name clashes.
The reason for using defines instead of spelling out "namespace SyncEvo"
is twofold:
1. if that should ever become necessary, it is easier to
rename the namespace via configure options by changing
the define
2. editors don't indent the whole file content
Install head files to a standard path, the remaining dependencies are
synthesis and boost
client-test is portable when ENABLE_MODULES is defined, no longer link to
backends libraries.
Add --enable-developer-mode, in which mode the backend scan path will be
under current build directory for development purposes.
The main motivation for this change is that it allows the implementor
of a backend to choose the implementations for the different aspects
of a datasource (change tracking, item import/export, logging, ...)
independently of each other. For example, change tracking via revision
strings can now be combined with exchanging data with the Synthesis
engine via a single string (the traditional method in SyncEvolution)
and with direct access to the Synthesis field list (now possible for
the first time).
The new backend API is based on the concept of providing
implementations for certain functionality via function objects instead
of implementing certain virtual methods. The advantage is that
implementors can define their own, custom interfaces and mix and match
implementations of the different groups of functionality.
Logging (see SyncSourceLogging in a later commit) can be done by
wrapping some arbitrary other item import/export function objects
(decorator design pattern).
The class hierarchy is now this:
- SyncSourceBase: interface for common utility code, all other
classes are derived from it and thus can use that code
- SyncSource: base class which implements SyncSourceBase and
hooks a datasource into the SyncEvolution core;
its "struct Operations" holds the function objects which
can be implemented in different ways
- TestingSyncSource: combines some of the following classes
into an interface that is expected by the client-test
program; backends only have to derive from (and implement this)
if they want to use the automated testing
- TrackingSyncSource: provides the same functionality as
before (change tracking via revision strings, item import/export
as string) in a single interface; the description of the pure
virtual methods are duplicated so that developers can go through
this class and find everything they need to know to implement
it
The following classes contain the code that was previously
found in the EvolutionSyncSource base class. Implementors
can derive from them and call the init() methods to inherit
and activate the functionality:
- SyncSourceSession: binds Synthesis session callbacks to
virtual methods beginSync(), endSync()
- SyncSourceChanges: implements Synthesis item tracking callbacks
with set of LUIDs that the user of the class has to fill
- SyncSourceDelete: binds Synthesis delete callback to
virtual method
- SyncSourceRaw: read and write items in the backends format,
used for testing and backup/restore
- SyncSourceSerialize: exchanges items with Synthesis engine
using a string representation of the data; this is how
EvolutionSyncSource has traditionally worked, so much of the
same virtual methods are now in this class
- SyncSourceRevisions: utility class which does change tracking
via some kind of "revision" string which changes each time
an item is modified; this code was previously in the
TrackingSyncSource
After a http request is sent out, while the client is waiting for response,
the server found client is unreacheable (client side network down temporarily or
intermidate router down), so the server stops sending and closed the tcp
conenction, causing a half closed connection at client side.
To fix this, client side will timeout (60 seconds) during wating and will
resend previous request to retry. After 3 failed retries, it will finally
give up leading to an abort. The timeout and retry count can be customized
in server config file.
The Synthesis Engine doesn't count these items because Synthesis wants
the engine to count only items deleted by the remote side. For our
statistics this looks a bit unrealistic (items before sync + added - deleted
does not add up to items after sync), so we now track items deleted
locally and use that number instead of the Synthesis statistics for deleted
items when we know that it is not complete.
In the command line, we show detailed statistics. In the GUI, it shows
up as part of the locally applied changes.
The new testComplexRefreshFromServerSemantic test covers this particular
aspect of the statistics.
This test checks whether the server a) creates no duplicates
during a slow sync (also covered by testManyItems) and b) correctly
associates the client's item with its own copy.
As described in Bugzilla #4703, the ScheduleWorld server currently
has a problem with the later (for contacts) and fails to delete
the item on the client, apparently because it doesn't remember the
LUID correctly. It also fails this test for text because it creates
duplicates.
The only way that I found not to execute a test was not to register
it in CPPUnit. FilterTest() does this by replacing a valid test
or test group with a dummy one, SkipTest, which just prints the
test name and that it is skipped.
Registering tests has to be intercepted at multiple levels:
- CPPUNIT_TEST in test suites
- ADD_TEST in ClientTest
- addTest in ClientTest
Not currently intercepted are complete test suites (CPPUNIT_TEST_SUITE).
The main purpose of this patch is to avoid running the time consuming
suspend and interrupt tests, but this feature might also be useful for
other tests, which is why it was implemented in a more general way.
- when libsoup is used (as on Moblin), the configure script sets
a default search for the SSLServerCertificates option which works
on Debian and Red Hat:
/etc/ssl/certs/ca-certificates.crt:/usr/share/ssl/certs/ca-bundle.crt
- with libcurl the library's own default is used; SSLServerCertificates
is empty, but can be set to override the default
- libsoup enables checking when either SSLVerifyServer or SSLVerifyHost
is on (does not support more control over checking), refuses to work
if no CA certificates file was found
- libcurl supports both strict checking (both options on) and less
strict checking (only check certificate, but not hostname)
Without that information, the wrong files are compared, leading to false
test failures. Because getItems() is also called indirectly via import(),
this change ripples through several functions.
When working with the Funambol engine, 0 meant "default values",
which were reasonably large for message size and unlimited for object
size.
After migrating to Synthesis and enabling these limits again, 0 has
no special meaning and is rounded up to some very small numbers. These
limits were too small for some items, therefore the settings used by
tests by default were bumped up to 128KB for message size (should
keep the overall message number fairly small) and 1GB for individual
items (basically unlimited again).
Config::testcases_server had no useful purpose. The extra dump()
method became redundant because the EvolutionSyncSource API *always*
returns "native" items and thus the generic method works fine.
During a refresh-from-client sync, the server has to delete all items
that it currently has before adding those sent by the client. This
new test covers that. It complements the testRefreshSemantic test,
which did that for the other direction and thus was renamed to
testRefreshFromClientSemantic.
The server also needs to tell other clients to delete the items
that were deleted this way during a refresh-from-client. This is
not currently tested.
As with the previous change from AGPLv3 to LGPL v2.1, this is covered
by copyright ownership and/or the Funambol contributor agreement.
The motivation for adding v3 is greater flexibility regarding
reusing the code in other places and relicensing it.
Comparison was missing altogether. Items were not imported as text/calendar
because the type wasn't set. Creating new items didn't work because <<REVISION>>
was in the wrong place: it needs to be in the DESCRIPTION, so that SUMMARY and first
line of DESCRIPTION remain identical (expected by some peers and our own sync source).
update-copyright.sh can be used to add copyright remarks for the current
year. It finds the authors who made a change in each file and adds/updates
their copyright remark. Intel employees are grouped under "Intel Corporation".
This is permitted by the Funambol contributor agreement (copy
is in doc/Sync4jContribution.pdf). It reduces the license mix
and allows distributing synccompare without special precautions.
Evolution vcard21 source defines testcases/vcard30.vcf as its native
test case file. For vcard21::testConversion we must use vcard21.vcf instead
because that is the format used for encoding the items.
Note that the test doesn't pass because synccompare does not support
old-style vcard and vcalendar encoding: line breaks aren't handled
correctly etc.
Removed the "encoding" source option. It was of dubious usefulness
with Funambol and has meaning with Synthesis at all.
Removed dead code in EvolutionSyncConfig.
Added "enableWBXML" sync option. WBXML is used by default, but for
debugging disabling it might be useful: dumping WBXML doesn't seem
to work reliably.
Most Client::Sync tests use the default encoding, usually WBXML unless
changed via CLIENT_TEST_XML=1. Client::Sync::*::testItemsXML always
uses XML and Client::Sync::*::testItems always WBXML.
The conversion test checks whether information is lost and/or modified
when converting the test items into the field list defined in the Synthesis
XML config. For that it converts into the field list and back, then compares
against the original test item file with synccompare.
Because a valid Synthesis session is necessary, this test hooks into
running a sync session via a callback that is triggered by the first
Synthesis progress event. It then aborts the session without contacting
the server.
boost::bind() with a reference should have worked (?), but didn't:
the bound function did not update the variable it was bound to.
Using a pointer worked.