Commit Graph

107 Commits

Author SHA1 Message Date
Jonas Smedegaard 693ba4c3fe Import Upstream version 1.5.3 2021-09-29 23:01:46 +02:00
Patrick Ohly 405ca882eb libsyncevolution: load backends in function instead of constructor
Previously, backends were loaded in a constructor. At some point (*), that
loading crashed with a segfault in the dynamic linker. To make debugging a bit
easier and rule out non-determinism as the root cause of that crash, loading
backends was moved into SyncContect::initMain().

(*) the crash happened when there were two backends which couldn't
be loaded due to missing libraries and was related to error strings.
A simpler test program did not trigger the problem, and now SyncEvolution
has suitable backends for (hopefully?!) all platforms, so it no longer
occurs, at least not during automated testing.
2016-09-26 12:58:26 +02:00
Patrick Ohly 2d5c60f644 SyncSource: add operation signal handler return code
This allows signal handlers to affect the operation execution and
result without having to throw an exception, which would get logged as
error and is not desirable when the operation gets skipped
intentionally. Needed for skipping SaveAdminData in the next patch.
2014-09-10 12:06:52 +02:00
Patrick Ohly dec741ea50 VirtualSyncSource, MapSyncSource: implement m_isEmpty
This is relevant for isUsable() and not so much (not at all?) for
slow sync prevention.
2014-09-08 11:07:31 +02:00
Patrick Ohly 8ac69096e8 command line: revise usability checking of datastores
When configuring a new sync config, the command line checks whether a
datastore is usable before enabling it. If no datastores were listed
explicitly, only the usable ones get enabled. If unusable datastores
were explicitly listed, the entire configure operation fails.

This check was based on listing databases, which turned out to be too
unspecific for the WebDAV backend: when "database" was set to some URL
which is good enough to list databases, but not a database URL itself,
the sources where configured with that bad URL.

Now a new SyncSource::isUsable() operation is used, which by default
just falls back to calling the existing Operations::m_isEmpty. In
practice, all sources either check their config in open() or the
m_isEmpty operation, so the source is usable if no error is
enountered.

For WebDAV, the usability check is skipped because it would require
contacting a remote server, which is both confusing (why does a local
configure operation need the server?) and could fail even for valid
configs (server temporarily down). The check was incomplete anyway
because listing databases gave a fixed help text response when no
credentials were given. For usability checking that should have
resulted in "not usable" and didn't.

The output during the check was confusing: it always said "listing
databases" without giving a reason why that was done. The intention
was to give some feedback while a potentially expensive operation
ran. Now the isUsable() method itself prints "checking usability" if
(and only if!) such a check is really done.

Sometimes datastores were checked even when they were about to be
configure as "disabled" already. Now checking such datastores is
skipped.
2014-09-08 11:07:31 +02:00
Patrick Ohly 237a240a7f SyncSource: allow marking databases as read-only
This is relevant for WebDAV database scanning (read-only
databases should not be the default) and could also be used
to enhance automatic setups (for example, do not use two-way
syncing for read-only databases).
2014-05-02 16:43:52 +02:00
Patrick Ohly 02088c22e0 SyncSource: add source name to all exception handling
Not all exceptions thrown while executing a source operation contain
the source name. SyncSource::throwError() does that, but SE_THROW() as
used in helper code does not. It is better to add the source name as
logging prefix. The other advantage is that all lines will have the
prefix, not just the first one.

The SyncSource operations need access to the SyncSource class which
contains them to access the display name of the SyncSource instance.
A positive a side effect of telling the wrappers about the instance at
construction time is that the caller no longer has to pass the source
reference.

This change allows removing the SyncSource::throwError() special
cases, which completes the transition towards having correct source
code location information for all exceptions.
2014-05-02 16:43:52 +02:00
Patrick Ohly de8461f802 code restructing: Exception, throwError()
Raising exceptions via throwError() looses the original source code
location information. Fixing that by explicitly passing that
information as additional parameter, created with the preprocessor
macro SE_HERE.

Several files included the complex SyncContext.h only because needed
throwError(). A better place for the revised implementation is the
Exception class which used to be in util.h, now Exception.h.

Simplifying the include statements exposed indirect include
dependencies and removed "using namespace std" from the compilation of
some source files which happened to rely on it. We want to get rid of
that name space polution, so fix the code instead of adding it back.
2014-05-02 16:43:52 +02:00
Patrick Ohly 27af370ac2 WebDAV: do not mangle UID when sending items
The WebDAV backends contained a hack where the UID inside the data was forced
to be identical to the resource name. This is wrong for items created by us
via POST (because the server may choose a resource name != UID) or by some
other entity (where we have no idea how the resource name got chosen).

This commit removes the hack. Testing must be updated to pass correct data
with the same UID as on the server when updating an item, because the backend
will no longer ensure that and changing the UID of a resource gets rejected by
some servers.

The hack was introduced for peers which do not store the UID (for example, a
vCard or iCalendar 1.0 based SyncML client). A better solution must be found,
probably involving the Synthesis engine and its read/update/write cycle.
2014-05-02 16:43:44 +02:00
Patrick Ohly fc8caadd8d PBAP: Suspend/ResumeSync() (FDO #72112)
The information that the sync is freezing is now also handed down to
the transport and all sources. In the case of PBAP caching, the local
transport notifies the child where the PBAP source then uses Bluez
5.15 Transfer1.Suspend/Resume to freeze/thaw the actual OBEX transfer.

If that fails (for example, not implemented because Bluez is too old
or the transfer is still queueing), then the transfer gets cancelled
and the entire sync fails. This is desirable for PBAP caching and
Bluetooth because a failed sync can easily be recovered from (just
start it again) and the overall goal is to free up Bluetooth bandwidth
quickly.
2014-04-01 16:45:09 +02:00
Patrick Ohly 3c01a1ebf6 PIM: enhanced progress notifications (FDO #72114)
This adds GetPeerStatus() and "progress" events.

To detect DB changes as they happen, the SyncSource operations are
monitored. Upon entry, a counter gets increased and transmitted
through to the PIM manager in syncevo-dbus-server using extended
SourceProgress structs (only visible internally - public API must not
be extended!). This will count operations which fail and count those
twice which get resumed, so the counts will be too high
occasionally. That is in line with the API definition; because it is
not exact, the API only exposes a "modified" flag.

Progress is reported based on the "item received" Synthesis event and
the total item count. A modified libsynthesis is needed where the
SyncML binfile client on the target side of the local sync actually
sends the total item count (via NumberOfChanges). This cannot be done
yet right at the start of the sync, only the second SyncML message
will have it. That is acceptable, because completion is reached very
quickly anyway for syncs involving only one message.

At the moment, SyncContext::displaySourceProgress() holds back "item
received" events until a different event needs to be emitted. Progress
reporting might get more fine-grained when adding allowing held back
events to be emitted at a fixed rate, every 0.1s. This is not done yet
because it seems to work well enough already.

For testing and demonstration purposes, sync.py gets command line
arguments for setting progress frequency and showing progress either
via listening to signals or polling.
2014-04-01 16:45:09 +02:00
Patrick Ohly 0855a8b961 core: mark error throwing or exit functions as SE_NORETURN
throwError() never returns normally. fatalError() terminates the process.
2014-01-17 16:15:16 +01:00
Patrick Ohly 7d12c0a586 read-ahead: tell SyncSource about the upcoming read accesses
Trying to predict in the SyncSource which items will be needed is
hard. It depends what the item is retrieved for (sync or
backup/printing) and on the sync mode (during a sync).

Instead of guessing, better have the user of the source tell the
source in advance what it will read. In most cases this is cheap
because it does not involve copying of items luids ("read all items",
"read new or modified items"). The source then can use its own internal
mechanisms to figure out what that means.

Only extracting specific items specified on the command line and the
backup operation must explicitly pass an ordered list of luids. For
the sake of simplicity, do that in a std::vector even though it
involves copying.
2013-07-10 13:07:44 +02:00
Patrick Ohly 6c551167c4 engine: change tracking optional for caching mode and item modification
Recording the revision of items during a caching sync is unnecessary
because the next sync will not use the information. For item
modifications, the information was not even recorded.

Now a "don't need changes" mode can be set in SyncSource, which is
done for caching and command line item operations. This is better than
second-guessing the mode in which a source is used.

SyncSourceRevision checks that flag and skips updating the
luid->revision mapping if set.

Individual backends can also check the flag and skip calculating the
revision to begin with.
2013-07-05 17:44:10 +02:00
Patrick Ohly ebef300ae5 SyncSource: support asynchronous add/update in utility classes
SyncSourceSerialize uses ITEM_AGAIN plus a callback in
InsertItemResult to indicate that and how an insert (aka add or
update) operation must be continued.

TrackingSyncSource merely passes that result through until it finally
completes.

Direct, raw access to items as done by the command line and restore
operations is not done asynchronously. The backend does not need to
know how it is used, the SyncSourceSerialize wrapper around the
backend will flush and wait.

Asynchronous deleting is not supported. It would require throwing an
exception or an API change (better - exceptions are for errors!)
because removeItem() has no return code which could be extended.
2013-07-05 17:44:09 +02:00
Patrick Ohly 856295b1e1 SyncSource: optional support for asynchronous insert/update/delete
The wrapper around the actual operation checks if the operation
returned an error or result code (traditional behavior). If not, it
expects a ContinueOperation instance, remembers it and calls it when
the same operation gets called again for the same item.

For add/insert, "same item" is detected based on the KeyH address,
which must not change. For delete, the item local ID is used.

Pre- and post-signals are called exactly once, before the first call
and after the last call of the item.

ContinueOperation is a simple boost::function pointer for now. The
Synthesis engine itself is not able to force completion of the
operation, it just polls. This can lead to many empty messages with
just an Alert inside, thus triggering the "endless loop" protection,
which aborts the sync.

We overcome this limitation in the SyncEvolution layer above the
Synthesis engine: first, we flush pending operations before starting
network IO. This is a good place to batch together all pending
operations. Second, to overcome the "endless loop" problem, we force
a waiting for completion if the last message already was empty. If
that happened, we are done with items and should start sending our
responses.

Binding a function which returns the traditional TSyError still works
because it gets copied transparently into the boost::variant that the
wrapper expects, so no other code in SyncSource or backends needs to
be adapted. Enabling the use of LOCERR_AGAIN in the utility classes
and backends will follow in the next patches.
2013-07-05 17:44:09 +02:00
Patrick Ohly c4de1cb227 SyncSource: simplify getPre/PostSignal()
Const-casting the member references is easier (= less code) because we
have a typedef for them and we can omit the access via "this".
2013-07-05 17:44:09 +02:00
Patrick Ohly 521c7ac967 PIM: allow removal of data together with database removal (part of FDO #64835)
There is a difference in EDS between removing the database definition
from the ESourceRegistry (which makes the data unaccessible via EDS)
and removing the actual database. EDS itself only removes the definition
and leaves the data around to be garbage-collected eventually. This is
not what we want for the PIM Manager API; the API makes a stronger
guarantee that data is really gone.

Fixed by introducing a new mode flag for the deleteDatabase() method
and deleting the directory of the source directly in the EDS backend,
if requested by the caller.

The syncevolution command line tool will use the default mode and thus
keep the data around, while the PIM Manager forces the removal of
data.
2013-05-29 09:13:38 +02:00
Patrick Ohly b5befe6cbf Logging: remove usage of Logger instance
Passing an explicit Logger instance to the SE_LOG* macros was hardly
ever used and only made the macros more complex.

The only usage of it was in some backends, which then added a prefix
string automatically. The same effect can be achieved by passing
getDisplayName(). Exception handling no longer needs a Logger instance,
the prefix alone is enough.

The other intended usage, avoiding global variables for logging by
passing a logger known to the caller, was not possible at all.

To make prefix handling more flexible, it is now passed as a "const
std::string *" instead of a "const char *".
2013-05-06 16:28:12 +02:00
Patrick Ohly 2894f5c573 SyncSourceBlob: avoid crash in ReadBlob()
Klocwork correctly reported that the underlying implementation in
libsynthesis must not be called with NULL pointers. Therefore always
allow it to set the return values, even if our caller was not
interested.
2013-03-19 14:58:46 +01:00
Patrick Ohly bfe1b062cb testing: more workarounds for Google CalDAV + unique IDs
Google became even more strict about checking REV. Tests which
reused a UID after deleting the original item started to fail sometime
since middle of December 2012.

To fix this, tests must differentiate reuse of an item by adding a suffix
("-A", "-B", etc.). If CLIENT_TEST_UNIQUE_UID has a value >= 2, that suffix
will be used when mangling the input item. Otherwise the suffix is ignored
and nothing changes.

For testing Google, CLIENT_TEST_UNIQUE_UID=2 is used.
2013-02-26 12:03:45 +01:00
Patrick Ohly f1bbe9802a SyncSource: add API for creating and deleting databases
The API supports creating a database with a specific name and UID. To
avoid ambiguities, deleting is done via the UID.

Looking up the UID for a "database" property value is meant to be
supported by opening the SyncSource and then getting the current
database that was opened.

This allows implementing a delete operation that uses the same
database lookup method as syncing.

All of the new methods have stubs which reject the operation
respectively tell the caller that no information is available. This
was done to avoid having to adapt all derived SyncSources. Support for
the new functionality is optional.
2012-10-25 16:43:46 +02:00
Patrick Ohly 53b72a035a InitState: merged InitState and InitStateClass
The new InitState template works for both classes and generic types.
The difference is in the kind of base class that it inherits from: the
one for classes "is" an instance of the wrapped type, the other "has"
such an instance. boost::is_class is used to pick the right utility
class, something that had to be done manually previously.

While touching the classes, the explicit initialization with "const
char *" was replace with a more general template constructor.

The main advantage is that other template code, like the D-Bus traits,
no longer needs to distinguish between InitState and
InitStateClass. The InitState implementation did not become
smaller. At least it avoids code duplication (m_wasSet handled in one
place).
2012-10-25 16:43:45 +02:00
Patrick Ohly 43371ac180 sources: force slow sync when listAllItems() returns no revisions
First, clarified that a source has to set all revision strings or
none. This was implied, but not spelled out explicitly.
Second, use that to detect when a sync must be done in slow mode.
Third, avoid calculating changes when that isn't needed. This last
point was left as TODO while in release preparations and can
be committed now.

This changes is needed for the PBAP backend which does not currently
set revision strings and thus cannot be used in incremental syncing.
2012-08-31 15:00:25 +02:00
Patrick Ohly 2673b67981 testing: fixed testInsertTwice
Backends need to declare how they'll react to inserting
the same item (= same UID) item twice.
2012-07-10 09:37:24 +00:00
Patrick Ohly 3716ae2c9a core, WebDAV: improved support for aborting while sleeping
When waiting for resending a failed message, the sleeping couldn't be
interrupted when using the D-Bus server. It now uses the SuspendFlags
infrastructure and glib, which detects abort requests sent via D-Bus
in addition to those sent via signals (which already worked earlier).
2012-06-20 12:26:43 +02:00
Patrick Ohly 5014f9b105 backends: merge with incoming data by default
All backends except for EDS replaced local data wholesale with an
incoming update, even if that update came from a peer which did not
store all properties. The EDS backend had already been configured
earlier to always merge remote and local data before writing it back.

Instead of updating each backend individually it makes more sense to
make the more intelligent (and more expensive) merging the default in
backends derived from SyncSourceSerialize/TrackingSyncSource. Backends
which do not want it can remove
"<updateallfields>true</updateallfields>" from
SynthesisInfo::m_datastoreOptions.
2012-06-15 12:25:52 +02:00
Patrick Ohly e749586310 testing: allow backends to register tests after main()
Backends are now allowed to create RegisterSyncSourceTest instances
with empty name. They will be ignored except that their init() method
will be called after main() and before the list of source tests is
used.

This can be used to create additional RegisterSyncSourceTest instances
at a time when the test of libsyncevolution is initialized. This is
needed by the WebDAV backend, which has to instantiate sources and use
the config system. That crashed when done before the ConfigProperty
instances were initialized.
2012-06-15 12:25:52 +02:00
Patrick Ohly c1a705169d testing: added Client::Source::*::testLinkedSources
The WebDAV backend must support different kinds of items in the same
collection. The new testLinkedSources covers this by adding, updating
and deleting an item of one kind and checking that other sources
referencing the same database do not see these changes.

This test must be activated for a specific source by adding links to
other sources using the same database. A separate commit will do that
for WebDAV.
2012-06-15 12:25:52 +02:00
Patrick Ohly a899633531 SyncML server: fixed invalid memory access in blob handling
Some parameters were not initialized before calling the underlying
libsynthesis implementation.
2012-05-22 09:16:32 +00:00
Patrick Ohly 314f073638 local + remote sync: negotiate UID support via SyncCap (BMC #22783)
This uses the new libsynthesis support for adding and checking entries
in the SyncCap to detect per datastore whether UID/RECURRENCE-ID are
truly globally unique and thus can be used to finding pairs. The
presence of the property alone is no guarantee for that.

Previously this kind of pairing was enabled only for local sync, which
was a hack which didn't work for local backends which didn't support
UID (for example, Maemo 5 calendar). It also didn't work for mixtures
of datastores with and without that kind of support.

"1122583000" was randomly chosen as pseudo sync mode. It is a number
because strings confuse Funambol. Note that SYNCMODESUPPORTED() only
works inside the compare script.
2012-05-03 09:45:36 +02:00
Patrick Ohly 5eed5ea60b testing: enhanced DAV source testing + infrastructure
The main goal is to test CalDAV/CardDAV sources as part
of a SyncML client and/or server. A test involving syncevo-http-server
is now named "<client storage><server storage>":
- edsfile = EDS in client, file in server (used to be syncevohttp)
- davfile = CalDAV/CardDAV in client, file in server (new)
- edsdav = EDS in client, CalDAV/CardDAV in server (new)

For this, WebDAVSourceRegister.cpp must be able to create test sources
which match the client 1/2 sync configs. The client "1" or "2" strings
are passed through the abstract ClientTest into the source A/B create
callbacks.  WebDAVSourceRegister.cpp cannot get this directly from
ClientTest because it lives in a plugin which is not necessarily
linked against ClientTest.

A conceptual change is that CLIENT_TEST_EVOLUTION_PREFIX/USER/PASSWORD
no longer override existing config properties. That is necessary
because the shared prefix is too simplistic for WebDAV (needs full URL
in "database"); also helps KDE (needs resource URI). The env variables
and the default "SyncEvolution_Test_" value for the database prefix are
still used if the config does not exist. That is useful to prevent
accidentally running client-test against the default databases.

The nightly setup script might (should!?) be made public to simplify
configuring a server.

Another change is the user-configurable part of client-test now lives
entirely in the _1/_2 client sync configs and contexts. From there the
source properties are copied into the Client::Source context each time
client-test runs.
2012-04-23 11:03:32 +00:00
Patrick Ohly 12334eb676 SyncSource: remove special RegisterSyncSource::InactiveSource pointer
The special semantic of the former RegisterSyncSource::InactiveSource
(invalid pointer of value 1) caused bugs, like using it in
--print-databases (=> segfault) or not being able to store the result
of a createSource() directly in a smart pointer (=> potential leak in
SyncSource::createSource()).

Obviously a bad idea to start with. Replaced with a
RegisterSyncSource::InactiveSource() method which creates a real,
inactive SyncSource instance which can and must be deleted by the
caller.

This is a SyncSource API change for backend developers.
Instead of RegisterSyncSource::InactiveSource, return
RegisterSyncSource::InactiveSource().

Comparisons against RegisterSyncSource::InactiveSource needs
to be replaced with a call to the new SyncSource::isInactive().

User visible fixes:
* --print-databases: no longer crashes when EDS or KDE backends
  are not usable. Instead it prints "not enabled during compilation or
  not usable in the current environment".
* --print-databases: continues with other backends even if
  one backend throws an exception, as the KDE backend does
  when it cannot find Akonadi. Error messages are printed.
2012-03-09 07:25:11 +00:00
Patrick Ohly b5410acbd9 SyncSource: rewrote callback handling for operations
A few operations had a callback registration mechanism, but not
all. For testing multi-cycle syncing and injecting concurrent changes
while a sync runs it is necessary to add code for several more
operations (begin data read, add/modify/delete item).

Instead of adding more custom code, this commit replaces the
home-grown callback lists with a boost::function wrapper which
contains pre- and post-signals based on boost::signals2.

Also simplifies the Synthesis DB Plugin implementation by moving the
common code (null check, exception handling) into the wrapper.

This removed the LOCERR_OK return code from some plugin
implementations (end data read, start data write). Implementations
returning LOCERR_OK must be provided by the SyncSource. Done by
SyncSourceSession (used by TrackingSyncSource) automatically.
2012-03-06 14:03:22 +01:00
Patrick Ohly 4c270e4e80 SyncSource: support multiple sync cycles in the same session
The state of SyncSourceChanges and SyncSourceRevisions must be reset
in beginSync() to get change tracking right when the same session
includes more than the traditional single sync cycle.

SyncSourceRevisions::initRevisions() must clear its set of revisions
before filling the data structure again. Otherwise items removed as
part of a sync are not detected as deleted in the next cycle.

Any backend derived using these classes (like any backend derived from
TrackingSyncSource) should work now and all other backends should be
fixed to support multiple cycles, so the <canrestart> flag is set
unconditionally.

Still need Client::Source tests for this expected behavior.
2012-03-06 14:03:21 +01:00
Patrick Ohly 00912e9e18 SyncSource + SyncContext: added requestAnotherSync()
The long term goal is that using a specific sync source once more in
another, new sync session can be requested per source.

The actual implementation however is per-session, using the new
libsynthesis "restartsync" session variable. This functionality
sits in SyncContext, because it is the one which has access to
the session variables.
2012-03-06 14:03:21 +01:00
Patrick Ohly 26ebe2f299 SyncSource: grant control over "read-only" mode
The Synthesis engine already supports read-only datastores. All writes
for such a store are silently discarded. Now a SyncSource can select
that mode by setting SynthesisInfo::m_readOnly to true in its
getSynthesisInfo() implementation.
2012-03-06 14:03:20 +01:00
Patrick Ohly 031ae896f5 testing: delay creating expected additional VEVENTs for Exchange
They are now created in callbacks instead of doing it in advance
for all cases.

Improves startup time (creating all of the artificial VEVENTs
took considerable time, in particular under valgind) and overall
resource consumption (because now only the next events is
kept in memory).
2011-12-08 13:59:22 +00:00
Patrick Ohly 26f31f5be6 testing: added Source::*::testRemoveProperties
This is primarily for ActiveSync where the test failed until support
for removing properties was added to activesyncd. Is also applied to
all other sources, just in case.

The EDS contact backend needs to keep the X-EVOLUTION-FILE-AS property
because EDS keeps adding it, which makes it impossible to test its removal.
2011-11-28 16:08:00 +00:00
Patrick Ohly 00de4dfa87 config: return value + "was set" for each config property
At the property level, the isDefault retval exposed whether the
property value was set explicitly in the config or taken from the
property default. That information got lost at the
SyncConfig/SyncSourceConfig level although there are cases where that
is relevant (like providing better error messages, BMC #23783).

Now that level uses the new InitState classes instead of plain
int/bool/std::string return values. Code which assigns these return
values to local variables doesn't need to be adapted. Directly using
the return value in an expression might need some work (typically
adding a get() if the compiler cannot infer the desired
type). Overriding the virtual methods always needs to be adapted.
2011-11-28 10:18:19 +01:00
Patrick Ohly 7d22805634 testing: added testLinkedItemsSubset for ActiveSync
This new test primarily covers the creation of artificial parent
events for detached recurrences in the activesyncd (BMC #22831). The
activesyncd must be run with EAS_DEBUG_DETACHED_RECURRENCES=1 set, so
that the artifical parent event is returned to the client. Otherwise
SyncEvolution cannot verify the correct creation of that event.

Putting this test into ClientTest.cpp instead of the ActiveSync
backend was done because these test cases may also serve as a useful
stress test for other backends. At the moment, they are only enabled
when CLIENT_TEST_SERVER is "exchange" or "apple". The artificial event
is only expected for "exchange".

A full test run with all imaginable combination of events would be
very slow. Therefore the tests choose roughly five starting points
scattered over the year 2012 and limit the number of events for each
test to five, except for the initial sequence with no events skipped:
that stores all events.

The the moment the test is known to fail with Exchange for those
weekly events which include the Sundays of the summer/winter time
change. The server uses incorrect UTC times in the UNTIL
clause. Perhaps the time zone definition that we send to it is not
quite correct?
2011-11-17 09:29:08 +01:00
Patrick Ohly ba289c899f phone sync: delete<->delete conflict + phone calendar+todo sync (BMC #23744)
When deleting an item on phone and locally, the next sync fails with
ERROR messages about "object not found". This has several reasons:
- libsynthesis super data store attempts to read items
  which may or may not exist (triggers ERROR message)
- it checks for 404 but Evolution backends only return a generic
  database error (causes sync to fail)

It turned out that ReadItem and DeleteItem are expected to return a
404 status when the requested item does not exist. This patch documents
that (only in the TrackingSyncSource, though), adds tests and fixes
EDS, WebDAV, file and sqlite backends accordingly.

This patch also suppresses the 404 error logging inside DeleteItem(),
while still returning that error code to the Synthesis engine. Not
logging that particular situation is consistent with the previous
SyncEvolution behavior of silently returning successfully when there
wasn't anything to delete.

In addition, more recent libsynthesis versions also no longer do
a ReadItem() call to test for existence. That would still trigger
a spurious (albeit now harmless) ERROR message.
2011-11-14 21:17:02 +01:00
Patrick Ohly 1cd49e9ecd CalDAV: revised RECURRENCE-ID -> EXDATE support
The previous approach (updating the internal cache) had the drawback
that X-SYNCEVOLUTION-EXDATE-DETACHED was also sent to the CalDAV
server. The work of generating it was done in all cases, even if not
needed. Found when running the full test suite.

Now the X-SYNCEVOLUTION-EXDATE-DETACHED properties are only added to
the icalcomponent that is generated for the engine in
readSubItem(). There's still the risk that such an extended VEVENT
will be stored again (read/modify/write conflict resolution), so
further changes will be needed to filter them out.

To ensure that this change doesn't break the intended semantic of
X-SYNCEVOLUTION-EXDATE-DETACHED, the presence of these properties is
now checked in the LinkedItems::testLinkedItemsParentChild test.
2011-11-12 12:50:09 +00:00
Patrick Ohly 14416b8deb testing: improved logging + CLIENT_TEST_SOURCE_DELAY
Exceptions reported by CppUnit only contained one file+line pair. If
that location was called multiple times inside a larger test, then it
was impossible to tell where it was called. The new assertion macros
and in particular CT_ASSERT_NO_THROW() solve this problem by catching
exceptions, adding the current file+line information and then
rethrowing an extended exception. When CppUnit finally logs the
problem, it will contain a complete call stack.

For this to work, every single line which might throw an exception
must be wrapped in a macro. Entering and leaving the line is logged
together with the wrapped expression as part of the test .log file.

doSync() is handled as a special case and gets the file+line info of
its caller via parameters.

New logging macros are introduced and used in LocalTests::testChanges:
instead of writing comments, call the logging macros and the string
will appear also in the .log file of the test.

Further areas for improvements:
- use CLIENT_TEST_LOG() everywhere
- reduce file names to just the base name is logged
- convert .log file into HTML with links into session logs and
  ClientTest.cpp source file
2011-11-09 10:37:05 +00:00
Patrick Ohly 741d55e8bf testing: renamed LinkedItems tests, added "no ID" variants
Numbering Client::Source::LinkedItems_xxx with xxx being a number is
confusing, in particular when the same number stands for different
test data. Now each set of linked items has an additional, unique name
which is used for Client::Source::LinkedItems<Name>.

Done in combination with adding more linked item tests and slightly
reorganizing the logic for adding them:
- a default set with VTIMEZONE is added in all cases
- some SyncML servers override that default set
- others, in particular peers accessed via their own backend,
  enable additional Client::Source tests on a case-by-case basis

Exchange is only tested with its own default set (with "Standard
Timezone" as TZID) and the all-day recurring set (as before).

All other CalDAV servers are now also tested with the all-day
set (previously exclusive to Exchange) and local floating time (= no
TZID, new).

Google CalDAV can't be tested with local time because it converts such
events into the time zone of the current user. All-day events need
special test data because Google adds a time to the UNTIL clause
(http://code.google.com/p/google-caldav-issues/issues/detail?id=63).

synccompare also needs to ignore that Google adds a redundant VTIMEZONE
to the all-day test cases.

Finally, Client::Source tests for updating a child event (with and
without parent) without UID and RECURRENCE-ID inside the payload were
added. These properties are removed via text operations. The
expectation is that the source is able to add them back (if needed)
based on the meta information that it has about the existing item.

The file source is unable to do that. When using it in an HTTP server,
the server will look to peers like a peer which doesn't support the
semantic (which indeed it doesn't) and thus the client will add back
the fields.
2011-11-04 11:04:38 +01:00
Patrick Ohly fbe71f0ef7 engine: allow StartDataRead/beginSync to start early and late, as needed (BMC #22881)
The unconditional change of calling StartDataRead (aka beginSync)
early enough so that ActiveSync can force a slow sync had negative
consequences, because now it was called before the peer was contacted
and credentials were accepted:
- broke the "sync started successfully" logic, resulting in
  notifications for syncs which were supposed to be retried silently
  (showed up in TestSessionAPIsDummy.testAutoSyncNetworkFailure)
- database dumps were done even if not needed because sync never
  starts

Now all backends are called as before unless they explicitly ask for
the early call. The ActiveSync backend does that. The downsides of
that approach do not matter much because syncing will start okay and
dumping of data is typically disable on that side of a local sync.
2011-10-20 15:18:48 +02:00
Patrick Ohly 1421769db8 SyncSource: allow StartDataRead/beginSync() to force a slow sync
This is needed for ActiveSync, which can detect inside beginSync()
that the sync key became invalid and then has to fall back to a slow
sync.

It is done by asking the engine to call StartDataRead earlier (via
<plugin_earlystartdataread>) and documenting the special 508 error
code. Depends on support for both in libsynthesis, thus the bumped
version number of it.

Normally SyncEvolution is meant to never throw an exception unless
something fatal happens, so it would have been better to add a return
code to beginSync(). But that is a fairly intrusive change, so instead
the simpler "use exception" solution is used. The change can still be
made later on.
2011-10-17 13:17:58 +02:00
Patrick Ohly 765ed0f833 TestingSyncSource+CalDAV: added "delete all" API call
During testing, removing all items is done via a special call in
TestingSyncSource. It used to be an utility method which fell back to
the normal SyncSource API.

This patch changes several things
- Reversed order in which items are deleted in that utility method,
  because removing children (= longer IDs) first tends to be supported
  better by servers (bug in CalDAV server, but still...).
- Allow backends to implement their own removeAllItems().
- Implement that in CalDAV + MapSyncSource as removing the merged
  items directly, instead of using a sequence of PUT+final DELETE.

Found while testing with Bedework CalDAV server. Makes testing more
robust and efficient.
2011-10-17 13:17:40 +02:00
Patrick Ohly 8a640d2ae2 SyncSource API: support and use Synthesis DB_DataMerged/Replace/Conflict result codes (BMC #22783)
Returning 207 = DB_DataMerged when an item replaced an existing one
wasn't correct (the data wasn't really merged) and also wasn't handled
as intended by the Synthesis engine. Now a backend can properly report what it did:
- data fully replaced
- data was merged
- data needs to be merged by engine

The last option is used by EDS and CalDAV when an add<->add conflict
is detected by the backends. In that case the Synthesis engine will
read the local item, merge it with the incoming item and write back
the result.

At the moment the tests assume that the more recent item wins
completely. But our field list config specifies a merge=lines mode for
some fields, which results in concatenating these fields in the merged
item. Thus the tests currently fail. Need to decide which behavior is
desired.
2011-09-14 15:07:30 +02:00
Patrick Ohly 9f10c2dd46 testing: cleaned up ClientTestConfig
The memset/memcpy of the embedded boost::function instances inside the
old ClientTestConfig was causing segfaults at the end of a client-test
run if compiled with optimization.

Therefore this commit turns ClientTestConfig into a proper class
containing members which initialize themselves (Bool wrapper class,
std::string), thus memset is no longer needed and used. Also added the
standard m_ prefix.

m_numItems is gone, was never set by any backend anyway and even
expected to be consistent in one test. Now CLIENT_TEST_NUM_ITEMS is
read by defNumItems() each time it is needed.

Removed "const char *" strings from method parameters. This revealed
that config.itemType (a const char *) was incorrectly passed to
insert() where the boolean "relax" parameter should have been given.
Replaced by "false" (= strict checking) even though the old code
must have run with an implicit "true" (= relaxed checking). Let's see
whether any tests fail now.

(cherry-picked from commit 6399bd8181)
2011-09-13 08:43:23 +00:00