First step towards removing "type". The parsing of the type string was
moved into SourceType. m_localType and comments were added to
illustrate how SourceType will be used in the future (not enabled
yet). getSourceTypeString() was removed because it was only needed for
error messages and will no longer be correct once "type" property is
gone.
SyncConfig inherited "const char *" from the Funambol C++ API and some
other methods used the same approach for efficient access to plain
strings. However, this has the disadvantage that dynamically generated
strings cannot be returned. SyncConfig had to use an awkward
workaround with a local string cache.
This patch converts most of that code to a normal std::string return
value and removes the string cache.
Out-of-tree backends must be adapted, otherwise they won't compile.
Some users have complained about getting duplicated contacts. The
exact reason is not known (no useful logs provided yet), but it might
be due to using "duplicate" as resolution strategy during slow syncs.
This causes slightly different contacts to be duplicated instead of
merging the two copies. This patch switches to "newer-wins", which
enables merging if both sides have time stamps. An even more
aggressive duplicate avoidance strategy would be "server-wins"
or "client-wins", but that should better be configurable.
Made ItemCache available as part of SyncSource.h and moved the "add
item to cache" code from SyncSourceRevisions into
ItemCache::backupItem(). The code turned out to be useful when
implementing backup in a backend which couldn't quite use
SyncSourceRevisions directly.
During local sync names like "addressbook" are no longer unique,
because they may exist in both the local and the remote context. This
patch introduces a "display name" composed from context and source name,
like this: "@<context>/<source>" (@default/addressbook).
The context is only used if needed, which currently is the case during
a local sync.
Changing the SyncSourceBase::getName() result was also considered, but
several places expect this to be the name of the source inside its
context, so an explicit SyncSourceBase::getDisplayName() turned out
to be safer.
MapSyncSource uses a MapConfigNode which shares the revision between
multiple uids. Updating the revision once it was seen the first time
broke change detection for the other items sharing that value.
Backends like XMLRPC need information about URL, proxy and SSL
settings, etc. This can be done via source specific properties, like
evolutionsource, but this is not how this is normally done. It would
be nicer if the exising per-peer properties could be used. The goal is
that a normal peer configuration can be created from a template with
the necessary information to enable sources using that information.
This patch makes this possible by adding a context parameter to
SyncSourceParams:
* @param context Additional non-source config settings.
* When running as part of a normal sync, these are the
* settings for the peer. When running in a local sync,
* these settings come from the "source-config" peer
* config inside the config context of the source.
* Testing uses "source-config@client-test". On the
* command line, this is the config chosen by the
* user, which may or may not have peer-specific settings!
Note that this still doesn't solve the problem for XMLRPC to SyncML
peer sync, because in that case ("normal sync") the context will be
the one describing the peer. SyncURL is already used and proxy
settings might not match.
The XMLRPC backends therefore was not changed and continues to use
evolutionsource.
The command line, like a lot of other code, used the escape/unescape
code in SafeConfigNode. For historic reasons, that code used ! as
escape character, which is awkward for the command line, because
that is a special character.
Instead of further overloading the SafeConfigNode, this patch moves
the escape/unescape code into its own utility class. This is the
cleaner approach anyway. It also adds unit testing for the code.
All other users of the old code are updated. Care must be taken here
to not accidentally switch to a different escape mechanism, because
the mechanism must remain compatible with the old implementation.
This is required when creating the configuration with virtual sources,
otherwise a virtual SyncSource will always be marked as 'disabled' because
it thought there is "no database to synchronize".
entry2mapid() escapes special characters of the local ID before
composing the key. mapid2entry() unescaped the whole key, before
splitting it. This is not quite symetric. It didn't cause problems in
practise though, because escape/unescape don't modify the -%x suffix.
This patch changes the order anyway. It also uses the slightly more
efficient rfind() instead of find_last_of() to locat the suffix.
With our own server, the testInterruptRetryServerAdd case failed. It
turned out that SyncSourceAdmin::mapid2entry() was generating " 1" for
empty remote ID and flags == 1, which then was stored as "1" and
misinterpreted as remote ID "1" with flags == 0 by
SyncSourceAdmin::entry2mapid().
Our config nodes cannot store leading or trailing spaces. We could
quote empty strings, but that would lead to backward compatibility
problems. Therefore this patch treats a single word in the value as
the flags and explicitly creates such values, instead of relying on
the key/value store to do that implicitly.
Implement blob support. Blobs are stored in the per-peer source
directory in a ".cache" sub-directory. We could have relied on the
peer name and device name to make file names unique when sharing
a common directory. Perhaps that is indeed the better solution:
~/.cache/syncevolution/blobs with logdir specifying the root?
Blob support is only needed when running as server. When running as
client, the binfile layer provides the necessary implementation.
Support for suspend was off by default because <resumesupport> was not
set. Adding it, because we should have whatever is
needed. <resumeitemsupport> depends on the ability to store blobs.
Not sure whether this is relevant on the client side, where these
services are provided by the binfile layer.
When automatic backups are enabled (the default), SyncSourceRevisions
and derived classes like TrackingSyncSource called listAllItems()
twice, once during backup and once while checking for changes.
This patch introduces caching of the result returned by the first
call. During a normal session, that will be during backup, with
change detection reusing the information. If backups are off, the
call will happen during change detection, as before.
Sharing of the listAllItems() result with change tracking only
works for the backup that is created at the start of a sync.
This piece of information is passed to the backend as part of
the BackupInfo structure.
The advantages of this change are:
- more efficient, in particular for sources where listAllItems() is slow
- avoids a race condition (backup stores one set of items, changes
are made, change detection works with another set)
That race condition is both fairly theoretical (short time window) and
had no impact on the correctness of the sync, only on the output (data
comparison would not show all changes synced later).
Now the race condition still exists if changes are made while a sync
runs, which is the bigger problem that still needs to be solved (for EDS,
see MB #3479).
Restoring data also calls listAllItems(). It does not use the cached
information, just in case that it is to be called more than once per
instance, and because there is no benefit.
As Jussi found out, the CheckSource() D-Bus method always declared
a virtual source as "unusable".
The underlying reason was that instantiating a SyncSource corresponding
to a virtual source config was not possible. If that had worked, it
wouldn't have checked the underlying sources.
This patch fixes both problems by extending VirtualSyncSource (now
instantiates and opens underlying sources) and
SyncSource::createSource() (now recognizes "virtual" and creates a
VirtualSyncSource).
Because creating proper sources depends on the SyncConfig, this must
be passed as parameter into createSource/VirtualSyncSource when
that kind of checking is desired.
Besides opening the underlying sources, VirtualSyncSource::open()
also verifies that it has a proper data format.
I noticed that this code had been copied from SyncSourceSerialize. I
moved the common code into a new SyncSourceBase::getDataTypeSupport().
Error checking also was a bit streamlined. For example, bad or missing
MIME type are detected in SyncSourceBase::getDataTypeSupport(), which
allows us to remove the corresponding check in SyncContext.cpp.
Note that SyncSource::throwError() should be used for source-specific
problems instead of SE_THROW(), because it includes the name of the
source in the exception.
With SHA-256, getting the same hash for different files
is very unlikely, so we can compare hashes instead of file
content to identify identical files.
This patch makes database dumps more space-efficient by reusing
existing files with the same content via hard links. "Same content"
is identified via the rather simple Hash() function, which
avoids reading the old files. We should better use a stronger
hash function or check the actual content?!
If nothing has changed, the only space required on disk for another
dump is for the directory structure and the .ini file.
Another advantage is that synccompare can be changed so that it
avoids including identical files in the comparison.
Note that this patch does not "trust" the revision strings. If it did,
reading the file content could have been replaced by lookup via
luid+rev. The advantage of checking and using the item content is
that changes in how the item is formatted show up properly in
comparisons. Improvements in that formatting lead to better backups,
which would not be the case when reusing old files just because the
revision has not changed.
This is the API change which will allow a SyncSource implemenation
to reuse data and files from a previous backup. Right now, neither
the caller nor the callee actually use this option.
MAKE/PARSETEXTWITHPROFILE() is used when converting data for or
from a backend. Properties can be enabled/disabled in that
conversion by enabling a "remoterule".
That rule used to be hard-coded as "EVOLUTION". With this patch,
the default is to use no rule. The Evolution backend
sets "EVOLUTION" in its base class.
SyncSource::createTestingSource() has to create sources using a
default configuration. It did that by using the "testing" config name.
Since introducing shared configs, that config happened to use shared
source properties from the "default" context.
This broke testing on my laptop after setting "evolutionsource" for a
specific peer in that context. Now "testing@testing" is used as config
name, which should be unique again.
For local test scenarios, it is possible the mapping table is never loaded but
flushed during endSession, this will empty the mapping file uncontentionaly.
According to synthesis, mapentries have type (TMapEntryType), for a server side
implementation, it is possible to have multiple mapentries with the same localid
and differeciated by entry type (such as tempidmap and pendingmap).
"x-calendar" was a typo.. However, to be backward compatible we will
accept both "x-vcalendar" and "x-calendar".
The configuration example is change to "x-vcalendar" to educate new users
to use "x-vcalendar".
The case of property names does not matter. A map where the key
is case-insensitive is thus a better data structure for storing
a set of property key/value pairs.
Such a type was already used by FilterConfigNode. This patch
cleans up the config nodes so that all of them use the new ConfigProps
type. The previous typedefs inside ConfigNode and FilterConfigNode are
preserved to keep old source code working without changing it.
The intention is to use the result of readProperties() directly as
parameter for setConfigFilter().
This patch introduces code changes for the new layout without actually
using it yet. Therefore all existing tests for the older layout
still pass. The new meaning of the former "server name" is introduced:
- A plain string now refers to a peer configuration (can be client
or server).
- The @ sign allows selecting a specific context. Different contexts
have independent sets of local sources and peer definitions.
- An empty peer name selects a view on the configuration which contains
no peer-specific properties. Not fully implemented yet.
The FileConfigTree is instantiated with a root which is one level high
up compare to before this patch (for example,
"~/.config/syncevolution" instead of
"./config/syncevolution/scheduleworld") and then config files include
that dropped level in their relative path name
("scheduleworld/config.ini" instead of "config.ini"). This allows
accessing the global properties in
"~/.config/syncevolution/config.ini" and will be used to move peers
further down the hierarchy
("~/.config/syncevolution/peers/scheduleworld/config.ini").
To keep the output of "--print-servers" consistent, the FileConfigTree
gets another parameter which identifies the subset of the larger tree
that is referenced by this FileConfigTree instance.
One side effect of this change is that FileConfigTree instances are no
longer completely separate. Something to keep in mind when
instantiating SyncContext multiple times (MB#8006).
Code may no longer make assumptions in which config node a property is
stored. This is determined by the new getNode() calls based on the
property attributes (hidden, sharing). The new layout is represented as
a set of config nodes. Older layouts use the same set of nodes with
identical instances assigned to them, if they don't really have separate
files for each of them.
SyncSourceNodes no longer grants direct access to the nodes, to catch
code which incorrectly access a specific node directly. For the same
reason the name of the nodes changed.
Code which needs access to all hidden or all visible properties now
does this via a config node returned by getProperties(). Currently
this is identical to the underlying nodes. Once the new layout is
active, this node will act as a multiplexer which gathers properties
from all underlying nodes when reading and picks the right one when
writing.
The "change ID" parameter for sources has been obsolete for a while
and was removed.
Reorganized the property registration so that it is a bit easier
to see which properties are hidden and which use non-default sharing.
The default sharing is "no sharing".
Some other code was improved while touching it:
- removed useless visibility[] array in favor of a i != 0 check in
SyncConfig::copy()
- added default parameters to save/checkPassword() methods
- some constness definition changes
- Property::getProperty() is a virtual call which could only be
overloaded in one case because the constness was wrong; now
getProperty() always returns the string value and getPropertyValue()
some other kind of representation of it, depending on the
class.
- ConstSyncSourceNodes is based on SyncSourceNodes instead of duplicating
it, which simplifies the implementation.
The simplified SyncSourceAdmin API changed slightly: instead of passing
a pointer to the source's SyncSourceNodes, the default nodes are now
found via the SyncSource pointer. For callers this is a bit less
work and it is more general.
For SyncML Server not based on Synthesis binfile implementation,
"refresh-from-remote" need be stated explictely to ensure the datastore supports
this.
Starting with this commit, SyncEvolution implements the
Synthesis session authentication and device administration
itself. The goal is to have better control over these steps
and become independent of the SDK_textdb which implemented
this before.
The SDK_textdb works, but it has several limitations, among
them the inability to choose a password. It is also called
"demo" in the source code, which raises some doubts about
its stability.
Because the Synthesis DB interface is now used both for database
access and for session handling, the Module_CreateContext() function
can not always select a source as context. When used as session
plugin, the global context pointer is NULL.
The Synthesis engine calls us at various points to save information.
Because it is not exactly clear how long we can buffer this
information in memory, it is always flushed to disk immediately.
It seems the engine calls this for existing items and expects it
to succeed, in contrast to what sync_dbapi.h says. SDK_textdb
also doesn't seem to check for existance before writing, so do
the same.
'priority' values have different interpretations in vCalendar1.0
and iCalendar2.0. Add scripts to do this conversion
to make it consistent.
There are 4 places to handle this kind of conversion:
receiving from server, sending to server, reading from
client backend, and writing to client backend.
For client backend reading and writing, implement a default
behavior in m_afterReadScript and m_beforeWriteScript. Users
could override 'getSynthesisInfo' to customize its own behaviors.
Also users could use default scripts and append its own
implementations in m_afterReadScript and m_beforeWriteScript.
Note that it is possible to lose some information for
priority value.
SynthesisInfo has 2 members m_incomingScript and m_outgointScript
which are used for beforewritescript and afterreadscript.
This could make code confusing. Change their names
Previously, sync failed because the datastore configuration specified
no means of storing the admin data (single chunk of text and local ID/remote ID
mapping). This could have been added by configuring the SDK_textdb as
<plugin_admin_module>.
Instead this patch implements the functionality inside SyncEvolution,
using a new ".server.ini" config node for the mapping and an
internal "adminData" property for the admin data text chunk. This is
more natural because it keeps the data under our control.
Authentication is configured via <simpleauthuser/pw>. In contrast to
using the SDK_textdb, this allows choosing the username and password.
Ultimately this should also be done inside SyncEvolution, to avoid
writing the username/password into the XML (unsolved encoding issue)
and into log files (privacy issue).
The intention is to get rid of the historic and inconsistent
naming of some classes and their corresponding files:
* EvolutionSyncClient = class derived from Funambol's SyncClient,
* SyncEvolutionConfig = SyncEvolution's config
With the strict 'namespace SyncEvo' and the syncevo/ path prefix for
most header files it is no longer necessary to have "SyncEvolution" or
"Evolution" in the names. This patch thus renames as follows:
EvolutionSyncClient => SyncContext
EvolutionSmartPtr => SmartPtr
SyncEvolutionCmdline => Cmdline
SyncEvolutionConfig => SyncConfig
SyncEvolutionUtil => util
The former EvolutionSyncClient always had a role that went beyond just
running a sync, for example it also provided config access. With the
upcoming server support it also won't be just a client. Thus the new
name "SyncContext".
The 'syncevo/' prefix is used throughout the code now.
removed whenever the prefix made it clear that the file belongs
to SyncEvolution. This helps finding incorrect include paths.
Quotes should be used exclusively for SyncEvolution files which don't
have a specific prefix yet (test.h, config.h) to help identifying
them.
Added syncevo/declarations.h, which has
This is now used for all SyncEvolution source files, except
for the GTK UI, which is written in plain C. In the library
it helps to avoid name clashes.
The reason for using defines instead of spelling out "namespace SyncEvo"
is twofold:
1. if that should ever become necessary, it is easier to
rename the namespace via configure options by changing
the define
2. editors don't indent the whole file content
std::iostream.str() created a temporary std::string which was
destructed when these functions returned, therefore
the c_str() pointer returned by these functions was invalid.
Changed the return type so that a dynamically created std::string
can be returned. Moved the functions into SyncSource, because
conceptually they are related to SyncSource::createSource().
When evnironment SYNCEVOLUTION_BACKEND_DIR is set, we will scan this path
instead of built-ins.
Also adpat the backend search code to use ReadDir wrapper class for exception
safety.
Install head files to a standard path, the remaining dependencies are
synthesis and boost
client-test is portable when ENABLE_MODULES is defined, no longer link to
backends libraries.
Add --enable-developer-mode, in which mode the backend scan path will be
under current build directory for development purposes.
Patrick Ohly
Chen Congwu
Zhu, Yongsheng