LogRedirect is intentionally written around lossy UDP communication:
when system() starts a command that produces lot of output, the kernel
drops data instead of deadlocking us (system() command blocked in write,
our process blocked waiting for command).
When we know that a command might produce much output *and* we redirect
output, we need a more intelligent replacement for system(). This patch
introduces Execute() for that purpose.
It reuses much of LogRedirect code. LogRedirect has a new constructor
which creates the pipes and then handles the output like it normally
would, except that the pipes are Unix domain datagram sockets and the
instance itself doesn't modify the global state of the parent (doesn't
register itself as logger, doesn't touch FDs 1 and 2).
Unix domain datagram sockets seemed like a combination of a lossless
connection with data boundaries, which has the advantage that the
existing process() code can be reused. When experimenting with
LogRedirect originally, it was observed that domain sockets did
deadlock when writing more than reading.
The problem with this patch is that Unix domain sockets do not flag
that there is no writer anymore. Therefore LogRedirect::process()
cannot detect when it should return.
The original creating the transport and remembering to set the callback approach:
1) For ObexTransport, the callback need to be set inside
createTransportAgent, because the 'connect' process might timeout.
2) It is error prone to remember setting the callback after
creating each transport.
Now setTransportCallback is integrated inside createTransportAgent.
The client-test output was previously interrupted by "Synchronization
successful/failed" output. That confused the nightly testing. With
SyncContext updated so that it supports setting an output channel,
we can redirect into a string stream.
We could check against expected output. Right now, it is ignored.
Cmdline and SyncContext standard output should always go
through a variable that can be pointed towards the actual
output channel.
Cmdline already had m_out for that, but it wasn't used
everywhere. Fixed.
SyncContext now has setOutput()/getOutput() and that is
used in SyncContext.cpp. It is not set anywhere yet.
To catch incorrect use of cout or cerr inside SyncEvolution,
the SyncEvo namespace defines its own cout and cerr which cannot
be used like std::cout/cerr, thus triggering compiler
errors. Use "std::cout/cerr" when necessary.
commit 49296bca70 fixed some problems but missed
another point:
If the type used by sub syncsource does not specify the data format explictly,
we cannot decide whether the format the backend supports is compatable with the
format a virtual SyncSource selects. Thus no warning in this case.
Found while testing with Nokia N900, sending 'respURI' will confuse the phone
and thinking different 'respURI' corresponding to different devices/configs.
When running with an invalid http_proxy env variable, I got the
following error:
[INFO] Transport timeout after 1:00min
[INFO] SoupTransport Failure: http://127.0.0.1:9000/syncevolution via libsoup: Cancelled
[INFO] SyncContext: resend previous request #1
[INFO] SoupTransport Failure: http://127.0.0.1:9000/syncevolution via libsoup: Service Unavailable
[INFO] Transport giving up after 1 retries and 1:17min
First, INFO messages should not reference internal classes. "request
request. Changed this into "resend previous message, retry #1".
Second, it shouldn't resend after a timeout. The check for "this is
the first message" was missing in that case.
Third, the "retryInterval zero" check was missing from the timeout
handling (noticed while looking at the code).
Since SyncEvolution no longer resends the first message if it times
out, all "Retry" tests failed right at the beginning. Now the first
and only the first interrupted "Retry" run is allowed to fail.
Implement blob support. Blobs are stored in the per-peer source
directory in a ".cache" sub-directory. We could have relied on the
peer name and device name to make file names unique when sharing
a common directory. Perhaps that is indeed the better solution:
~/.cache/syncevolution/blobs with logdir specifying the root?
Blob support is only needed when running as server. When running as
client, the binfile layer provides the necessary implementation.
Support for suspend was off by default because <resumesupport> was not
set. Adding it, because we should have whatever is
needed. <resumeitemsupport> depends on the ability to store blobs.
Not sure whether this is relevant on the client side, where these
services are provided by the binfile layer.
'PreventSlowSync' was enabled, so that when a sync could not be conducted without
falling back to a slow sync, we will abort the sync and warn the user about this
unnormal condition.
However if the sync session is inited from a remote peer (e.g. Server Alerted
Sync), this kind of warning on our side has no way to convey to the remote peer;
thus causing remote peer confused about the sync failure. It is reasonable to
disable this feature in this kind of scenario because: 1) The remote peer might
not have 'PreventSlowSync' check capability and we must work with them 2) For
SyncEvolution server, this kind of checking should be implemented at server side.
Patrick found some valgrind warnings and put a prelimilary version
of part of the fix. Finished the work in this patch.
Added a libical timezone supression rule, with an additonal patch
in libsynthesis and the updated supression rules, valgrind is happy.
Implement new notification for auto sync according to
requirements. The main changes are messages description
and internationalized.
3 kinds of messages are defined and shown via notification
server: 'is syncing', 'Sync complete' and 'Sync problem'.
They are used in below scenarios:
1) sync is successfully started, show 'is syncing'
2) sync is ended, show 'Sync complete' or 'Sync problem'
3) sync has problem, once sync is not successfully started,
and a fatal error occurs, show 'Sync Problem'. If not a fatal
problem, don't show anything.
To detect when sync is successfully started and done, SessionListener
is defined to listen to these changes. This gives AutoSyncManager
opportunities to send notifications once these statuses are changed.
Two buttons 'Dismiss' and 'View' are added into all notifications
to let users dismiss the notification or pop up sync-ui.
They are implemented by optional 'actions' mechasim of notification.
So if the notification server doesn't support actions, don't add
them.
First, setting an emtpy value was not possible on the command
line because it wasn't among the valid values.
Second, the SAN generator tried 1.0 first when SyncMLVersion was empty.
It should have tried 1.2 first before falling back to 1.0.
Support generating a legacy version SAN package leverating the new
API in libsynthesis.
Added a new configuration option 'SyncMLVersion'(1.0|1.1|1.2), on
the server side, this is used to select the SAN version to send to
the client (default to empty which means it will first try SAN 1.2
and fallback to SAN 1.1 automatically). On the client side, this is
used to select the SyncML version talking to remote server, default
to empty which means it will try the latest commonly supported SyncML
version by the server, when setting a value the client will only try
the specifid version.
When the URI is empty in a client config, the Synthesis engine ignores
the source and reports a "no config" error when that source was the
only one. This patch adds a more specific error message.
All of our config templates contain the uri of each source, so this
only happens when configuring new sources manually.
When a single source is active and we abort because of an unexpected
slow sync, then the overall sync result was set to the internal 20048
"abort datastore" status. This status must be translated back to
the root cause of the abort, the 22000 "unexpected slow sync" error.
The right place to do this is before recording the status in the
SyncReport and before returning it. Any status read after that from
individual sources will already have 20048 replaced with the
right status.
The restore and status operations accessed the existing session
directory using the same code paths as running the session. Therefore
they ended up modifying the status code and start/end times.
This patch makes it more obvious what LogDir::startSession() does by
introducing a mode enum instead of a single boolean, which wasn't
enough. In SourceList, accessSession() replaces startSession() for
read-only access.
For convenience of people reading the status.ini, it had a string
describing the start/end time in local time with TZ offset. The minute
part of that was wrong, always printing the month instead of the
minutes.
Using numbers 1..9..10.. as suffix had the problem that sorting
puts -10 in front of -2. Because we use hh-mm as base name, this
limited us to 9 sessions per minute, which is easily reached
when doing local testing.
By switching to -a to -z as suffix, we get 27 sessions per minute
before moving on to less nice characters (possible, but might
make file names unusuable eventually). That should be enough.
In local tests I got as far as -q.
With this change, the LogDir expire tests can be merged back
into one function.
An additional benefit is that the suffix is visually different
from the base name, which makes it a bit easier to identify
yyyy-mm-dd-hh-mm manually.
As part of the recent changes for collision checks code was added
which does a ReadDir() on the log dir and throws and exception when
that doesn't exist. If the directory does not exist (now checked
explicitly), we don't have to check for collisions.
Set by SyncEvolution in status.ini before starting an sync. Replaced
with the final status code if the sync completes. Finding this code
here in a session report implies that the process responsible for the
session died unexpectedly, for unknown reasons.
Printing (and thus logging) information about "preparing" items is
currently not particularly useful because the Synthesis engine doesn't
really do much at that point. This will become different once we add
filtering, because then each item really must be retrieved and parsed.
This patch therefore temporarily disables the printing of that message.
The XML config is now only printed at logLevel >= 5, which saves
roughly 80KB. A minimal log is now at around 120KB, but I don't see
how I can make that smaller.
During development and in nightly testing, level 4 is suitable because
it avoids the XML dump but includes message translations and more
detailed logging from the Synthesis engine. It should also work well
for user bug reports. Level 5 adds verbatim message dumps and the XML
config dump.
First, the check was only done when the user forced a type. I'm
not sure why it was limited like that. IMHO alerting the user of
an inconsistent config would make sense also for other cases.
Second, it didn't work correctly because it compared against
getPeerMimeType(). This is not the type configured in the config, but
rather something hard-coded in the backend. For
"type=todo:text/x-vcalendar" it returned "text/calendar", which led to
a warning although the virtual "calendar+todo" also used
"text/x-vcalendar".
Third, the warning was only printed once and spelled inconsistently.
If both sub data sources are configured incorrectly, SyncEvolution
should complain about both before proceeding. It might also abort,
but that is debatable and not changed in this patch.
These tests cover the new priority-based implementation of
LogDir::expire(). Because our sorting breaks for more than 10 sessions
per minute (MB #9759), the test has to start anew in the middle with
a clean logdir.
The new algorithm is called after each sync, regardless whether that
succeeded or failed. The goal is to keep the number of session dirs
below maxLogDirs. If that is zero, all sessions are kept, as before.
The new algorithm will never:
- remove the last session, it is needed for reporting to the GUI
- remove the only session which has a dump of a specific source
Victims for removal are found by prioritizing all of them, evicting
oldest first in case of a bind. Sessions with database dumps are more
important than those without, sessions with changes (locally or on
server) more than those without changes, and finally sessions with
errors more than those without.
The useless INFO message about removing old sessions was removed. It
only confused users.
These tests cover the LogDir::haveDifferentContent()
utility function and the backup feature of sync sessions.
The latter is based on instantiating a SourceList with
FileSyncSources which can be pointed to different sets of
items via a symlink. This allows to simulate changes between
sessions as well as during a session.
The code was meant to ensure that whenever multiple session,
regardless of which peer, happen to fall on the same session
(unlikely, but happens in testing), an increasing sequence
number is attached to the new directory name.
This only worked if the current peer was among those who started in
that second, otherwise the collision check was avoided. In addition,
the code recycled sequence numbers of directories which had been
removed. So in the even more unlikely case of doing the following all
in one second, we ended up reusing the base name and thus had
an incorrect ordering:
- create foo-2010-01-31-12-00
- create foo-2010-01-31-12-00-1
- delete 2010-01-31-12-00
- reuse foo-2010-01-31-12-00
This found when adding unit testing of the revised expiry mechanism.
The underlying problem is that sorting of sessions depends on linear
time. Because we use local time for the directory names and clocks might
get reset, this is not guaranteed (MB #9759).
This utility function will be used to test the "shared file content"
backup feature and for more intelligent expiry of session dirs. It
depends on inodes and thus may return a false "have different content"
when dirs were copied. This is not going to happen normally and is
acceptable.
Delaying the database comparison was done initially for servers to
limit it to the sources really used by the client. In a client the
same delay makes sense for syncs which fail because the peer cannot be
reached. It is possible to delay the dump because starting the session
no longer depends on it thanks to the m_isEmpty() operation.
The current solution of checking for empty sources by looking at the
"before" database dump is a hack which fails if creating that backup
was disabled. It also prevents delaying the dump until after the
start of the session, because the information must be available
before starting the session, so that the XML config can be created
with slow sync check enabled if needed.
This patch adds a new operation which backends must implement if they
want to provide the "is empty" information for the slow sync
detection. This is optional. If the information is not available, slow
sync detection is always on, even when not strictly needed.
When automatic backups are enabled (the default), SyncSourceRevisions
and derived classes like TrackingSyncSource called listAllItems()
twice, once during backup and once while checking for changes.
This patch introduces caching of the result returned by the first
call. During a normal session, that will be during backup, with
change detection reusing the information. If backups are off, the
call will happen during change detection, as before.
Sharing of the listAllItems() result with change tracking only
works for the backup that is created at the start of a sync.
This piece of information is passed to the backend as part of
the BackupInfo structure.
The advantages of this change are:
- more efficient, in particular for sources where listAllItems() is slow
- avoids a race condition (backup stores one set of items, changes
are made, change detection works with another set)
That race condition is both fairly theoretical (short time window) and
had no impact on the correctness of the sync, only on the output (data
comparison would not show all changes synced later).
Now the race condition still exists if changes are made while a sync
runs, which is the bigger problem that still needs to be solved (for EDS,
see MB #3479).
Restoring data also calls listAllItems(). It does not use the cached
information, just in case that it is to be called more than once per
instance, and because there is no benefit.
When moving the slow sync setting into SyncSource, I didn't adapt
the lookup of the virtual sources. SourceList was inconsistent in
that it could only look up normal sources by name. Fixing this
inconsistency solves the sefault.
As suspected in the previous commit for this issue, the engine still
picks the normal <datastore> when asked for "./Calendar". This patch
renames, for example, "calendar" to "calendar+todo@calendar" to avoid
cluttering the global namespaces for URIs with sources that clients
should not synchronize against.
The "@" separator is used because it is already a reserved character
for config names, albeit not yet for source names. A slash "/" would
have been nicer because it cannot be part of a source name, but when
trying that the Synthesis engine treated the second part as URI
parameters (instead of part of the source name).
The Synthesis engine requires that each <datastore> has a unique
integer <dbtypeid>. This is part of the API for progress events
and configuring the <datastore>s.
SyncEvolution uses the source name as unique key and mapped that
to an integer via Hash(), without checking for hash collisions. This
patch fixes that by assigning an ID the same way the first time a
source is used. Hash collisions are checked at that point. Without
hash collisions (the assumption which had to hold already anyway),
the same IDs will be used as before.
The ID is recorded in the new internal "synthesisID", which means that
it won't change even when there are collisions and sources are added
and removed.
Another benefit of this patch is that the source lookup can be done
easily via the unique ID, which is useful for an upcoming patch (which
will rename some sources inside the Synthesis engine).
Because SourceList assigns the unique ID, it has to prevent that users
of it add sources, normal or virtual, without going through
SourceList. Therefore the m_virtualDS and vector base class were made
private.
This patch further clarifies what exactly the role of SourceList is
and that normal sources and virtual sources are handled differently.
This is important so that code can iterate over one or the other set.
The call was removed because
a) it is obsolete
b) because it is incomplete (doesn't cover virtual sources).
c) removing it allows us to change the SourceList implementation
(no need to expose vector)
When running with "calendar", "todo" both inactive and only activated
indirectly via "calendar+todo", the XML config listed <datatores>
called "calendar" and "todo" (with aliases, if specified) and the
"calendar+todo" <superdatastore>.
The N85 can start a sync with a config like that, but it sends events
and tasks to the "./Calendar" URI, which then end up the calendar
<datastore> instead of getting dispatched via the <superdatastore>.
First problem: setting an alias for <superdatastore> was
missing. Fixed in this patch.
Second problem: the engine might still pick the normal <datastore>
when asked for "./Calendar". Not fixed in this patch. Need to rename
sub-datastores?
Super datastores do not have a target entry in a binfile client's profile.
They have to be parameterized via their sub-datastores, using the
special <super>uri format for the sub-datastores' URI. "super" is
the name of the super datastore and "uri" its URI. The sync mode
is also taken from the sub datastores.
When an unwanted slow sync is detected for one or more normal sources
which are part of a virtual sync source, then the user must rerun
the sync with the virtual source named on the command line.
To implement this, a set of source names is more suitable than a list.
Changed SyncML::slowSyncExplanation() accordingly. The information
that a source is part of a virtual source is recorded in the new
SyncSourceReport::m_virtualSource. It is checked first before adding
the normal source's name to the set.
Saving the virtual source name in the status.ini and sending it via
D-Bus GetReport() is perhaps not necessary, but just in case it was
also added under the key "source-<source>-virtualsource = <virtual source>".
First, treat config errors as fatal. Previously the code tried to
continue somehow, but a) it was unclear whether that would have worked
(adding virtual source although its sub sources are missing?) and b)
running with a broken configuration might damage data (e.g. syncing
with wrong URI).
Second, some spelling fixes.
Third, added check that a normal source is not contained in two or
more different virtual sources.
With this patch, SyncML server sources are only opened() and their
data dumped when a client really uses them. As before, sources are
only enabled in the server if their sync mode is not "disabled". This
tolerates sources which cannot be instantiated because their "type" is
not supported.
The patch changes the SourceList and its methods so that they can do
the database dumps and comparisons for a single source at a
time. SourceList tracks which of its sources were dumped before the
sync and uses that information at the end to produce the "after sync"
comparison.
That "after sync" comparison was a reduced copy of the
dumpLocalChanges() source code. The copy was replaced with a suitably
parameterized call to dumpLocalChanges(), which became easy after
adding the "oldSession" parameter in a recent patch. That output now
is as follows:
-------------------------> snip <-----------------------------------
Changes applied during synchronization:
+---------------|-----------------------|-----------------------|-CON-+
| | LOCAL | REMOTE | FLI |
| Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| addressbook | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| two-way, 0 KB sent by client, 0 KB received |
| item(s) in database backup: 20 before sync, 20 after it |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| calendar | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| two-way, 0 KB sent by client, 0 KB received |
| item(s) in database backup: 20 before sync, 20 after it |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| start Wed Feb 10 16:38:15 2010, duration 0:02min |
| synchronization completed successfully |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
Data modified locally during sync:
*** addressbook ***
no changes
*** calendar ***
no changes
-------------------------> snip <-----------------------------------
Previously the last heading was "Changes applied to client during
synchronization", which is wrong for the server (it is not a
client) and did not properly distinguish between item and data
changes (items may be changed without affecting the set of data,
as in removing one item and adding it with the same content).
In a server, the "*** <source> ***" part is only printed for active
sources, whereas the table always contains all sources with sync mode
!= "disabled". If we had progress events for the server, it should be
more obvious that some sources were not really used during the
sync. Alternatively we could also remove them from the report.
Also fixed several other such "to server/client" messages. They were
written from the perspective of a client and were wrong when running
as server. Using "remotely" and "locally" instead works on both client
and server.
The ABORTDATASTORE() handling in the Synthesis engine was fixed
so that it aborts the whole session, without looking at further
datastores.
In order to get our desired behavior (continue checking other sources),
a special LOCERR_DATASTORE_ABORT error code was introduced. We use
that in our ABORTDATASTORE(), then translate it back to the SyncEvolution
specific STATUS_UNEXPECTED_SLOW_SYNC = 22000.
The reasons for keeping that code are:
- the sync-UI checks it
- encoding in the sync-UI that LOCERR_DATASTORE_ABORT == slow sync
would expose an implementation detail that we might want to change
in the future
The alias can be read directly from the current sync source, there
is no need to store it in SourceConfigSpecials first. After moving
the m_forceSlow flag into SyncSource, the whole SourceConfigSpecials
can be removed, which simplifies the code.
Using the <datastoreinitscript> has the advantage that a slow sync
chosen by the engine after <alertscript> was called because of an
anchor mismatch is now caught.
This patch also adds a reminder that the dependency on a database dump
should be removed and slightly modifies the check for slow sync. This is
done with an eye towards enabling this code on the server side (not done
yet).
The main motivation for this change was that access to that flag was
needed inside the SyncSource while experimenting with slow sync
detection.
In the end, the flag was not really needed there. Applying this patch
might make sense anyway, just in case that a similar need arises
again.
Eventually it might also allow to remove the entire
SourceConfigSpecials structure.
The syncevo-dbus-server never instantiated an ObexTransportAgent
in its own createTransportAgent() implementation because that code
hadn't been updated.
Instead of duplicating the code for instantiating the various agents,
now DBusSync::createTransportAgent() calls the underlying
SyncContext::createTransportAgent(). It passes its glib main loop
pointer and expects to get an agent which uses that.
It is a bit inconsistent that we have to pass a GMainLoop pointer
through an interface that is supposed to be independent of glib.
Therefore it is declared as "void *". When adding other main loop
integrations, we can either add more parameters or reinterpret the
content of that pointer.
This patch also cleans up the reference counting of the GMainLoop.
Now the reference is no longer handed over to the agent. That solution
was not entirely exception-safe. Instead the constructed agent has to
increase the reference count itself, or do whatever is necessary
instead to keep using the loop. In the ObexTransportAgent, a reference
to the right GMainContext is kept.
As Jussi found out, the CheckSource() D-Bus method always declared
a virtual source as "unusable".
The underlying reason was that instantiating a SyncSource corresponding
to a virtual source config was not possible. If that had worked, it
wouldn't have checked the underlying sources.
This patch fixes both problems by extending VirtualSyncSource (now
instantiates and opens underlying sources) and
SyncSource::createSource() (now recognizes "virtual" and creates a
VirtualSyncSource).
Because creating proper sources depends on the SyncConfig, this must
be passed as parameter into createSource/VirtualSyncSource when
that kind of checking is desired.
Besides opening the underlying sources, VirtualSyncSource::open()
also verifies that it has a proper data format.
I noticed that this code had been copied from SyncSourceSerialize. I
moved the common code into a new SyncSourceBase::getDataTypeSupport().
Error checking also was a bit streamlined. For example, bad or missing
MIME type are detected in SyncSourceBase::getDataTypeSupport(), which
allows us to remove the corresponding check in SyncContext.cpp.
Note that SyncSource::throwError() should be used for source-specific
problems instead of SE_THROW(), because it includes the name of the
source in the exception.
This patch adds more intelligent searching for the ".after" database
dump of each source that gets compared. This solves the problem that
if the last session involved only one source, comparison of all other
sessions was called "impossible".
The patch is careful to pick the last session which really involved
the source. This is not based on the existence of the dump, because
writing that might have failed or been disabled. Instead the status.ini
of the session is looked at. Note that this is different from the search
in dumpDatabases(), which can work with any dump of the source in
any peer at any time.
As a minor clean-up, some variables were named to reflect what
they really refer to nowadays (dumps are done in directories, not files).
This is one half of reusing old backup data. SourceList determines the
most recent backup made when syncing with any peer in the same
context. These peers share the same sources, therefore they can also
share backup data.
The "after" dump is checked first, because it is the more recent one.
The "before" dump of the same session is used when creating
the "after" dump.
This patch also prevents creating empty backup dirs when the backup
operation is not supported by the backend. Because all backends supported
this so far, this wasn't a problem.
When the context name contains characters which need to be escaped in
the logdir name (like "@client-test-1"), the getting all sessions
in that context failed.
The reason is that parseDirName() returns the peerName without unescaping
it. The context name check then compared the unescaped name against
the escaped one and not detecting the match.
IMHO parseDirName() should hide the escaping mechanism and return unescaped
strings, but this is the more intrusive patch and thus I don't change it.
In any case, something like this should be documented for the parameters
of the function.
This is the API change which will allow a SyncSource implemenation
to reuse data and files from a previous backup. Right now, neither
the caller nor the callee actually use this option.
getPreviousLogdir[s]() had the side effect of modifying the log dir
used by the instance. In many cases, the parameter was the one
from the SyncContext instance that the LogDir had access to.
This patch makes this a bit more obvious by using the SyncContext
log dir the default and only calling LogDir::setLogdir() explicitly
when needed.
A peer can be contacted via multiple transports, these urls are splitted via
space or tab.
If multiple urls are avaibale, when creating the outgoing connection, we will
use the first available one.
This patch replaces src/syncclient_sample_config.xml with a
combination of src/syncevo/configs/syncevolution.xml and the
config fragments that are shared with Synthesis upstream.
These fragments are installed in /usr/share/syncevolution/xml (or
the corresponding data path). From there they are read at runtime
to compose the final XML configuration. Users can copy individual files
into the corresponding directory hierarchy rooted at
$XDG_CONFIG_HOME/syncevolution-xml to replace individual fragments.
New fragments can be added there or in /usr/share.
For testing, these two directories can be overridden with the
SYNCEVOLUTION_XML_CONFIG_DIR env variable. No tests have been added
for this yet. There's also no documentation about it except this
commit message - add something to the HACKING guide once this
new concept stabilizes.
Developers can add new fragments in the source tree, invoke make and run the
resulting binary in client mode. As before, a complete config is included
in the binary. However, it is only sufficient for SyncML client mode.
For server mode, the files are expected to be installed (no need to maintain
a list of files in a Makefile for that) or SYNCEVOLUTION_XML_CONFIG_DIR
must be set.
At the moment, the following sub-directories are scanned for .xml files:
- the root directory to find syncevolution.xml
- datatypes, datatypes/client, datatypes/server
- scripting, scripting/client, scripting/server
- remoterules, remoterules/client, remoterules/server
Files inside "client" or "server" sub-directories are only used
when assembling a config for the corresponding mode of operation.
The goal of this patch is to simplify config sharing with Synthesis
(individual files are easier to manage than the monolitithic one), to
share files between client and server with the possibility to add
mode-specific files, and to allow users to extend the XML
configuration. The most likely use case for the latter is support for
more devices.
Previously, remote rules for the different devices listed in
syncserv_sample_config.xml were not used by SyncEvolution.
This patch moves the ZYB remote rule into a client-specific remote rule,
thus removing a complaint from libsynthesis about the unknown <client>
element when running as server.
Because we are using the unified upstream config, some parts of the config
have changed:
- There is a SYNCLVL field in all field list. This is currently unused
by SyncEvolution, but doesn't hurt either.
- A new iCalendar 2.0 all-day sanity check was added (for older Oracle servers?).
- The CATEGORIES defition in vBookmark was extended.
- some comment and white space changes
Because this is such fundamental change, extra care was taken to
minimize and verify the config changes. Here's the command which compares
old and new config for clients plus its output:
$ update-samples.pl syncevolution.xml client | diff -c -b syncclient_sample_config.xml -
***************
*** 31,42 ****
<scripting>
<looptimeout>5</looptimeout>
- <function><![CDATA[
- // create a UID
- string newuid() {
- return "syuid" + NUMFORMAT(RANDOM(1000000),6,"0") + "." + (string)MILLISECONDS(NOW());
- }
- ]]></function>
<macro name="VCARD_BEFOREWRITE_SCRIPT_EVOLUTION"><![CDATA[
// a wordaround for cellphone in evolution. for incoming contacts, if there is only one CELL,
// strip the HOME or WORK flag from it. Evolution then should show it. */
--- 30,35 ----
***************
*** 118,123 ****
--- 111,124 ----
}
]]></macro>
+ <function><![CDATA[
+ // create a UID
+ string newuid() {
+ return "syuid" + NUMFORMAT(RANDOM(1000000),6,"0") + "." + (string)MILLISECONDS(NOW());
+ }
+ ]]></function>
+
+
<!-- define script macros for scripts that are used by both vCalendar 1.0 and iCalendar 2.0 -->
<macro name="VCALENDAR_INCOMING_SCRIPT"><![CDATA[
***************
*** 145,150 ****
--- 146,158 ----
DTSTART = CONVERTTOUSERZONE(DTSTART);
MAKEALLDAY(DTSTART,DTEND,i);
}
+ else {
+ // iCalendar 2.0 - only if DTSTART is a date-only value this really is an allday
+ if (ISDATEONLY(DTSTART)) {
+ // reshape to make sure we don't have invalid zero-duration alldays (old OCS 9 servers)
+ MAKEALLDAY(DTSTART,DTEND,i);
+ }
+ }
// Make sure that all EXDATE times are in the same timezone as the start
// time. Some servers send them as UTC, which is all fine and well, but
***************
*** 265,275 ****
</scripting>
-
<datatypes>
-
<!-- list of internal fields representing vCard data -->
<fieldlist name="contacts">
<field name="REV" type="timestamp" compare="never" age="yes"/>
<!-- Name elements -->
--- 274,283 ----
</scripting>
<datatypes>
<!-- list of internal fields representing vCard data -->
<fieldlist name="contacts">
+ <field name="SYNCLVL" type="integer" compare="never"/>
<field name="REV" type="timestamp" compare="never" age="yes"/>
<!-- Name elements -->
***************
*** 680,689 ****
$VCARD_INCOMING_NAMECHANGE_SCRIPT
]]></incomingscript>
</datatype>
-
-
<!-- common field list for events and todos (both represented by vCalendar/iCalendar) -->
<fieldlist name="calendar">
<field name="ISEVENT" type="integer" compare="always"/>
<field name="DMODIFIED" type="timestamp" compare="never" age="yes"/>
--- 688,696 ----
$VCARD_INCOMING_NAMECHANGE_SCRIPT
]]></incomingscript>
</datatype>
<!-- common field list for events and todos (both represented by vCalendar/iCalendar) -->
<fieldlist name="calendar">
+ <field name="SYNCLVL" type="integer" compare="never"/>
<field name="ISEVENT" type="integer" compare="always"/>
<field name="DMODIFIED" type="timestamp" compare="never" age="yes"/>
***************
*** 787,793 ****
<subprofile onlyformode="standard" name="VTIMEZONE" mode="vtimezones"/>
! <!-- sub-profile for todoz -->
<subprofile name="VTODO" nummandatory="1" showifselectedonly="yes" field="ISEVENT" value="0">
<property name="LAST-MODIFIED" suppressempty="yes">
--- 792,798 ----
<subprofile onlyformode="standard" name="VTIMEZONE" mode="vtimezones"/>
! <!-- sub-profile for tasks -->
<subprofile name="VTODO" nummandatory="1" showifselectedonly="yes" field="ISEVENT" value="0">
<property name="LAST-MODIFIED" suppressempty="yes">
***************
*** 1394,1401 ****
<!-- non-standard properties -->
! <property name="CATEGORIES">
! <value field="CATEGORIES"/>
</property>
<property name="CLASS" suppressempty="yes">
--- 1394,1402 ----
<!-- non-standard properties -->
! <!-- inherit CATEGORIES from vCard 3.0, i.e. comma separated -->
! <property name="CATEGORIES" values="list" valueseparator="," altvalueseparator=";">
! <value field="CATEGORIES" combine=","/>
</property>
<property name="CLASS" suppressempty="yes">
***************
*** 1416,1435 ****
<use profile="vBookmark"/>
</datatype>
! <fieldlists/>
! <profiles/>
! <datatypes/>
</datatypes>
<clientorserver/>
-
- <client type="plugin">
- <remoterule name="ZYB">
- <manufacturer>ZYB</manufacturer>
- <model>ZYB</model>
- <!-- information to disable anchors checking -->
- <lenientmode>yes</lenientmode>
- </remoterule>
- </client>
-
</sysync_config>
--- 1417,1424 ----
<use profile="vBookmark"/>
</datatype>
!
</datatypes>
<clientorserver/>
</sysync_config>
The new feature of preventing problematic slow syncs has "problems" of
its own:
- the sync-UI still has to be adapted to deal with prevented slow syncs,
which might not happen in time for 1.0 beta
- we don't have user feedback about this deviation from previous
behavior
- the nightly testing runs into unexpected slow syncs with Memotoo
and Mobical, which do a slow sync when they have no data on the
server
This patch therefore introduces a configuration option
"preventSlowSync" which is off by default. This is also a good place
to explain the feature. Expert users and testers can enable the
feature per peer.
We can change the default value later on after some more testing. It
will be on without users having to change existing configs, because
default values are not set explicitly in configs.
When we enable this by default, the nightly testing must be
reconfigured to run with it disabled for Memotoo and Mobical (probably
best done explicitly inside client-test).
A dbus unit test is failed due to the change of getLogDirs. The problem
is the checking condition: sort the log names by their
creation time encoded in them when the peer name is empty, not the
prefix because the prefix contains the context information.
The reason was that the code which aborted the sync and prints the explanation
wasn't triggered here: the Synthesis engine stops all by itself because the
only active source failed.
I changed the printing of the explanation so that:
- it is printed when aborting, to explain the ERROR
that is printed there
- it is always (including the previous case) printed
again underneath the final status overview produced
by the command line
For this to work without code duplication, the formatting of the explanation
was moved into SyncReport. It kind of fits there because the message can be
generated for a SyncReport instance. For that case it would be nice if the
actual peer name used for that sync session was in the report; I didn't want to
make too many changes at once, though.
If the peer name is empty and context is given, we only match
the context and ignore the comparison of peer name for the
changes of encoding context information in the file name.
Currently for the given logdir, we compare the context of log
name and the given config name. This gives the capability to
extinguish same peer names for different contexts.If the behavior
is not to compare it, it's easy to change.
SyncContext::getServer() = m_server returns the string as passed in
by the user. This may be without the right context, if the peer name
alone is unique enough to select a config (scheduleworld2 instead
of scheduleworld2@other-context).
What we need is the normalized name as used by SyncConfig. Added
SyncConfig::getConfigName() for that and renamed getServer() to
getPeer(), with suitable documentation.
The normalized config name for peers in the default context is just
the peer name, without the @default. Using that makes the directory
names shorter and can be done with less code.
The comparison with peerName + @default can be removed, because that
can never happen when @default is not encoded in the name (?).
If these changes are okay, they should be squashed with the previous
patch before merging into master.
The Logdir has no info about context and is not normalized. This
will have problem when the peer name has arbitrary characters.
Normalize the peer name and adding context info in the log dir.
This could help find the corresponding config and look up the 'PeerName'
in the config for sync ui.
The OpenKeyByPath() and OpenSubKey() functions had to be called in
cases where the key might not exist, which triggered exceptions
(profile not created yet, end of datastore list, reading already
deleted session key).
This patch changes the functions so that the failure to read open
the key can (optionally!) be reported as a NULL SharedKey. The default
is to throw an error, as before. The locations where exceptions really
were triggered are changed so that they check for that, which also
simplifies the code.
When the overall source and thus sync session status is "OK" despite
non-zero item failures, the status is set to 22001 =
STATUS_PARTIAL_FAILURE. The command line then reports "some changes
could not be transferred (local, status 22001)".
The syncevolution command line return code indicates a failure just as
for all the other non-ok statuses.
For QA: it should be possible to reproduce this problem by setting the
backend of a source to the file backend and then pointing it towards
an existing directory where the current user has no write
permissions. Then do a "refresh-from-server" with a non-empty server.
When we can be sure that we have no local data in a source,
then the check for an unexpected slow sync can be suppressed
because there is no risk of duplicates or lost data.
The check depends on having done a backup before. If that was
disabled, the check is enabled because there is no other API
to query the content of the local database.
It would be nice if we had the same information about the server,
but that is not available a) without running the sync and b) not
sent by all servers.
This solution depends on ABORTDATASTORE() in an <alertscript>,
which is supported by libsynthesis only with an additional patch.
ABORTDATASTORE() is done for each source where the sync mode
requested by the server does not match the expected one.
The check is only added when a slow sync would not be acceptable.
Explicit "slow" and "resfresh-from-server" (which is done under
the hood as a slow sync?!) don't need the check. Distinguishing
unexpected "slow" from "refresh-from-client" has to be done
via ALERTCODE(), SLOWSYNC() returns true in both cases.
Note that currently, "no local data" and "first time sync" are
ignored when checking for acceptable slow syncs. This might have
to be added.
In addition, the code in SyncContext::doSync() is notified via the
"delayedabort" session variable that it should abort after processing
the current message and before sending the reply. This is necessary to
get the <alertscript> executed for all sources. Calling ABORTSESSION()
inside the script was tried first but aborted right away, so that an
unexpected slow sync could only be detected for one source.
Abort handling didn't work for aborting before sending a reply:
- SessionStep would return STEPCMD_SENDDATA
- we would start sending
- abort was checked later
A good point to abort a sync is after the engine has processed
a message and before sending out the reply. This patch adds that
check for STEPCMD_SENDDATA and checkForAbort() directly after
the SessionStep() call.
Because the whole suspend/abort handling is somewhat tricky, this
patch adds debug logging around those state changes.
The code which detects the request to abort also analyzes the sitution
and prints some guidance for the user how to recover from the problem:
[ERROR] Aborting because of unexpected slow sync for source(s): calendar
[INFO] Doing a slow synchronization may lead to duplicated items or
lost data when the server merges items incorrectly. Choosing
a different synchronization mode may be the better alternative.
Restart synchronization of affected source(s) with one of the
following sync modes to recover from this problem:
slow, refresh-from-server, refresh-from-client
Analyzing the current state:
syncevolution --status syncevolution_client calendar
Running with one of the three modes:
syncevolution --sync [slow|refresh-from-server|refresh-from-client] syncevolution_client calendar
On the server, doing this check in <alertscript> is not sufficient
because the server does the anchor check after executing the
<alertscript> and thus switches to an unexpected slow sync later on.
Need a different solution for server initiated syncs with phones.
This patch adds a plain text explanation of the Synthesis and SyncML
error codes. It explains whether the error was found locally or
remotely and includes the original status code, because sometimes
this is more helpful than the text.
The ERROR logging uses these explanations and the sync report
formatter.
This patch also adds some more status codes:
- 418 "already exists"
- 22000 "unexpected slow sync"
The latter is in the range reserved for application codes. We'll
use that for MB #2416.
Printing of per-source status codes was avoided, both at the point
where they were encountered as well as in the final sync report. The
reason for that was that often, the overall sync status is duplicated
for each source.
But for those cases were each source has a different status or no
error at all, printing that piece of information is useful, so this
patch adds it.
Note that now "no status code or error" for a source means that it
synchronized successfully.
A SyncML server cannot resend a message. With HTTP, a reply simply
cannot be sent, for other transports the D-Bus API doesn't support it.
Therefore RetryInterval is ignored and only RetryDuration is used as
the final timeout after which the session is aborted.
The same transport callback is used for client and server. It was
changed to log the timeout duration (sent in via its parameter). The
rest of the logic is in the caller of TransportAgent::wait().
While touching the timeout code in initSAN(), the resending of the SAN
message was removed. This makes the code consistent with the HTTP
SyncML client case, where the initial message is also not resent.
The new test-dbus.py TestConnection.testTimeoutSync covers thus timeout
handling. It depends on setting RetryDuration to 10 in the server-side
config of the sc-api-nat device.
Generate and send source statuses and progress to dbus clients.
For source modes in statuses, 'restore-from-backup' is sent once
a source is restored. For progress, the granularity is source level.
Thus, the progress is simply calculated based on the number of
sources and how many of them that have been completed.
To report source status and progress, SyncContext::restore
should have a mechanism to give out events to dbus server.
Here 'displaySourceProgress' is re-used. Some synthesis events are
simulated in restore. To follow up the way of 'displaySourceProgress',
a pseudo mode 'SYNC_RESTORE_FROM_BACKUP' is defined for source sync
mode. This is treated as a special sync mode only for restore from
backup.
Also add test code in testRestoreByRef to check source
status and progress.
The current use of recording the initial error ever printed during a
sync session is that it can be repeated at the end of the
"syncevolution" command line output or when printing sessions.
Initially the error is set to "synchronization process died prematurely".
Seeing that error in a session status.ini means that the report
wasn't updated because SyncContext::sync() never completed properly -
or is still running.
This patch also adds reseting of m_status in SyncReport::clear(),
which seemed to be missing.
"SyncReport buffer" was used as storage for later printing of the
report at the end of sync(). However, its scope ended earlier than
that. If the struct hadn't been entirely on the stack, we would have
had a problem.
With everything allocated on the stack, this wasn't a problem until I
added a std::string (next commit). Then it accessed a deconstructed
string => segfault.
sub datasources referenced by a super datastore will not be visible to user for
sync. The 'sync' and 'URI' properties is not set and will use the value set
in the super datastore.
The 'type' propery will be checked for consistency (values in sub datasoruce and
super datsource should be the same), if failed a warning will be generated and
continue the sync with the type specified by the super datasource.
When the config name is "" or "@xxx"(only with context the peer
name is empty), return all reports of different peers in the
specified logdir. They are sorted by their creation time,
which are encoded in their dir names.
Context info is not recorded in the logdir or directory name.
Normally different logdirs are used for different contexts to
distinguish them. If some contexts share the same logdir,
then all sessions are returned if only context is specified no
matter which context the sessions belong to.
The key 'peer' is also stored in every report dictionary sent to
dbus clients. To get it from the dir name, a new method
'getPeerNameFromLogdir' is added in the class 'LogDir'.
Also fix a bug of reports oder. Return reports in descending
order, which means the newest report at first.
Change the unit tests of test-dbus.py for ordering problem
and the peer name is empty. Add some new logging dirs to
test the ordering when the peer name is empty.
Step information 'waiting' is appended into status for the
progress spinner. The spinner should be disabled unless 'waiting'
is explicitly appended in the status.
'waiting' means the dbus server is blocked and waiting for something,
typically IO events, such as sending data, waiting for response. Once
it is absent from status, the dbus server is done with waiting.
Change corresponding unit tests in the test-dbus.py for statuses
have been changed.
After merging the "phone" branch, incoming SyncML messages
were no longer replied to properly. The code always ignored
the available message and replied with a SAN message. Now
the code checks for server mode *and* availability of incoming
data before generating a SAN.
move initSAN from sync() to doSync()
It was once designed to put initSAN as early as possible to reduce
unnecessary resource if client does not respond (automatic sync).
However it lost the signal handling. Let's move it to doSync() at
the moment and come out the other solution for automatic sync later.
Timeout handler was not registerd for the transport, fixed.
As Nokia phone stores calendar and todos in one datastore, we need to utilize
synthesis superdatastore to handle this case. A virtual SyncSource is introduced
which has a backend type "virtual" and its local database path points to a list
of sub syncsources. The virtual syncsource is maintained by SyncSourceList and
used to generate appropriate synthesis superdatastore configuraitons.
unescapeJoinedString is introduced to parse a comma seperated list of sub
syncsources. At this time, the configuration is fixed to work with
"calendar+todo" case.
Nokia phone (S40, 7710c) does not accept the null content type.
SyncSource::getPeerMimeType is added for this purpose, each backend
implementation can provide a MimeType used for sync by default. For
TrackingSyncSource based backend implementations it maps to getMimeType
directly.
Previously, the log file name was hard-coded inside libsynthesis and
changed between libsynthesis release (sysynclib_linux.html ->
sysynclib_uni_linux.html) without SyncEvolution being able to notice
or control this.
Now the file name is set by SyncEvolution. This depends on an
up-to-date libsynthesis with the "filename" configuration option.
commit 607be1a4da did not work as expected.
For the "strict aliasing" warning, it said:
"
Strict aliasing is an assumption, made by the C (or C++) compiler, that
dereferencing pointers to objects of different types will never refer to the
same memory location (i.e. alias each other.)
The exception to the rule is char*, which is allowed to point to any type.
"
So let's cast void* to char* instead of char* to void*.
This patch makes the configuration layout with per-source and per-peer
properties the default for new configurations. Migrating old
configurations is not implemented. The command line has not
been updated at all (MB #8048). The D-Bus API is fairly complete,
only listing sessions independently of a peer is missing (MB #8049).
The key concept of this patch is that a pseudo-node implemented by
MultiplexConfigNode provides a view on all user-visible or hidden
properties. Based on the property name, it looks up the property
definition, picks one of the underlying nodes based on the property
visibility and sharing attributes, then reads and writes the property
via that node. Clearing properties is not needed and not implemented,
because of the uncertain semantic (really remove shared properties?!).
The "sync" property must be available both in the per-source config
(to pick a backend independently of a specific peer) and in the
per-peer configuration (to select a specific data format). This is
solved by making the property special (SHARED_AND_UNSHARED flag) and
then writing it into two nodes. Reading is done from the more specific
per-peer node, with the other node acting as fallback.
The MultiplexConfigNode has to implement the FilterConfigNode API
because it is used as one by the code which sets passwords in the
filter. For this to work, the base FilterConfigNode implementation must
use virtual method calls.
The TestDBusSessionConfig.testUpdateConfigError checks that the error
generated for an incorrect "sync" property contains the path of the
config.ini file. The meaning of the error message in this case is that
the wrong value is *for* that file, not that the property is already
wrong *in* the file, but that's okay.
The MultiplexConfigNode::getName() can only return a fixed name. To
satisfy the test and because it is the right choice at the moment for
all properties which might trigger such an error, it now is configured
so that it returns the most specific path of the non-shared
properties.
"syncevolution --print-config" shows errors that are in files. Wrong
command line parameters are rejected with a message that refers to the
command line parameter ("--source-property sync=foo").
A future enhancement would be to make the name depend on the
property (MB#8037).
Because an empty string is now a valid configuration name (referencing
the source properties without the per-peer properties) several checks
for such empty strings were removed. The corresponding tests were
updated resp. removed. Instead of talking about "server not found",
the more neutral name "configuration" is used. The new
TestMultipleConfigs.testSharing() covers the semantic of sharing
properties between multiple configs.
Access to non-existant nodes is routed into the new
DevNullConfigNode. It always returns an empty string when reading and
throws an error when trying to write into it. Unintentionally writing
into a config.ini file therefore became harder, compared with the
previous instantiation of SyncContext() with empty config name.
The parsing of incoming messages uses a SyncContext which is bound to
a VolatileConfigNode. This allows reading and writing of properties
without any risk of touching files on disk.
The patch which introduced the new config nodes was not complete yet
with regards to the new layout. Removing nodes and trees used the
wrong root path: getRootPath() refers to the most specific peer
config, m_root to the part without the peer path. SyncConfig must
distinguish between a view with peer-specific properties and one
without, which is done by setting the m_peerPath only if a peer was
selected. Copying properties must know whether writing per-specific
properties ("unshared") is wanted, because trying to do it for a view
without those properties would trigger the DevNullConfigNode
exception.
SyncConfig::removeSyncSource() removes source properties both in the
shared part of the config and in *all* peers. This is used by
Session.SetConfig() for the case that the caller is a) setting instead
of updating the config and b) not providing any properties for the
source. This is clearly a risky operation which should not be done
when there are other peers which still use the source. We might have a
problem in our D-Bus API definition for "removing a peer
configuration" (MB #8059) because it always has an effect on other
peers.
The property registries were initialized implicitly before. With the
recent changes it happened that SyncContext was initialized to analyze
a SyncML message without initializing the registry, which caused
getRemoteDevID() to use a property where the hidden flag had not been
set yet.
Moving all of these additional flags into the property constructors is
awkward (which is why they are in the getRegistry() methods), so this
was fixed by initializing the properties in the SyncConfig
constructors by asking for the registries. Because there is no way to
access them except via the registry and SyncConfig instances (*), this
should ensure that the properties are valid when used.
(*) Exception are some properties which are declared publicly to have access
to their name. Nobody's perfect...
This patch introduces code changes for the new layout without actually
using it yet. Therefore all existing tests for the older layout
still pass. The new meaning of the former "server name" is introduced:
- A plain string now refers to a peer configuration (can be client
or server).
- The @ sign allows selecting a specific context. Different contexts
have independent sets of local sources and peer definitions.
- An empty peer name selects a view on the configuration which contains
no peer-specific properties. Not fully implemented yet.
The FileConfigTree is instantiated with a root which is one level high
up compare to before this patch (for example,
"~/.config/syncevolution" instead of
"./config/syncevolution/scheduleworld") and then config files include
that dropped level in their relative path name
("scheduleworld/config.ini" instead of "config.ini"). This allows
accessing the global properties in
"~/.config/syncevolution/config.ini" and will be used to move peers
further down the hierarchy
("~/.config/syncevolution/peers/scheduleworld/config.ini").
To keep the output of "--print-servers" consistent, the FileConfigTree
gets another parameter which identifies the subset of the larger tree
that is referenced by this FileConfigTree instance.
One side effect of this change is that FileConfigTree instances are no
longer completely separate. Something to keep in mind when
instantiating SyncContext multiple times (MB#8006).
Code may no longer make assumptions in which config node a property is
stored. This is determined by the new getNode() calls based on the
property attributes (hidden, sharing). The new layout is represented as
a set of config nodes. Older layouts use the same set of nodes with
identical instances assigned to them, if they don't really have separate
files for each of them.
SyncSourceNodes no longer grants direct access to the nodes, to catch
code which incorrectly access a specific node directly. For the same
reason the name of the nodes changed.
Code which needs access to all hidden or all visible properties now
does this via a config node returned by getProperties(). Currently
this is identical to the underlying nodes. Once the new layout is
active, this node will act as a multiplexer which gathers properties
from all underlying nodes when reading and picks the right one when
writing.
The "change ID" parameter for sources has been obsolete for a while
and was removed.
Reorganized the property registration so that it is a bit easier
to see which properties are hidden and which use non-default sharing.
The default sharing is "no sharing".
Some other code was improved while touching it:
- removed useless visibility[] array in favor of a i != 0 check in
SyncConfig::copy()
- added default parameters to save/checkPassword() methods
- some constness definition changes
- Property::getProperty() is a virtual call which could only be
overloaded in one case because the constness was wrong; now
getProperty() always returns the string value and getPropertyValue()
some other kind of representation of it, depending on the
class.
- ConstSyncSourceNodes is based on SyncSourceNodes instead of duplicating
it, which simplifies the implementation.
The simplified SyncSourceAdmin API changed slightly: instead of passing
a pointer to the source's SyncSourceNodes, the default nodes are now
found via the SyncSource pointer. For callers this is a bit less
work and it is more general.
SyncEvolution only uses one active engine instance otherwise the Synthesis
binfile maybe corrupted and leading to unexpected slow sync.
Add preSync() to carry out work needs to be done before we finally detect this
is a server session or client session. It initlize a tempory engine instance for
logging purpose and is destructed after leaving this function.
This removes code which was added as part of the some of the SAN
handling patches. The code added a second Synthesis engine while the
first one was still running, using this statement:
// reinitializes the engine, only at this time can we decide whether
// this is a server session or client session.
SwapEngine swapengine(*this);
I recognize the comment. It must have been copied from an older
SyncEvolution revision. However, SyncEvolution no longer uses two
instances of the engine, so this code can and has to be removed again.
Instead the first instance is reinitialized once the full config is
known.
The effect was that both the old and the new instance had the file
with the anchor data open. Then the more up-to-date data was overwritten
by the older one without a valid anchor when it was deconstructed,
leading to a slow sync in each session.
Throwing the exception about an unsupported transport must be
reached also for OBEX/Bluetooth URLs when ENABLE_BLUETOOTH is off.
Otherwise the function is left without valid return code.
Outgoing OBEX connection implementation, only binds over Bluetooth now.
Integrates with gmainloop so that the opertaions in the transport will not
block the whole application.
It uses Bluetooth sdp to automatically discovery the corresponding service
channel providing SyncML service; the process is asynchronous. Callback
sdp_source_cb and sdp_callback are used for this purpose. sdp_source_cb is a
GIOChannel watch event callback which poll the underlying sdp socket, the
sdp_callback is invoked by Bluez during processing sdp packets.
Callback obex_fd_source and obex_callback are related to the OBEX processing
(Connect, Put, Get, Disconnect). obex_fd_source is a GIOChannel event source
callback which poll the underlying OBEX interface, the obex_callback is
invoked by libopenobex when it needs to delivering events to the application.
Connect is splited by several steps, see CONNECT_STATUS for more detail.
Disconnect will be invoked when shutDown is called or processing in
obex_fd_source_cb is failed, timeout occurs or user suspention. It will first
try to send a "Disconnect" command to server and waiting for response. If
such opertaion is failed it will disconnect anyway. It is important to call
wait after shutdown to ensure the transport is properly cleaned up.
Each callback function is protected by a "Try-catch" block to ensure no
exception is thrown in the C stack. This is important otherwise the application
will abort if an exception is really thrown.
Using several smart pointers to avoid potential resource leak. After initialized
the resource is held by ObexTransportAgent. Copy the smart pointer to the local
stack entering a function and return to ObexTransportAgent if the whole
process is correct and we want to continue. First, it ensures the resource is
released at least during ObexTransportAgent destructing; Second, it can also
try to release the resource as early as possible. For example cxxptr<ObexEvent>
will release the resource during each wait() so that the underlying poll will
not be processed if no transport activity is expected by the application.
"SyncURL" is used consistently for the address of the remote peer to
contact with.
Use "peerIsClient" configuration property to decide whether this should be a
Server Alerted Sync Session or a client initated.
Use "remoteIdentifier" as the string sending to the remote peer during SAN so
that remote peer can identifier the server properly.
If this should be, will first generate and send SAN in SyncContext::doSync,
get the client request and inialize the session as server with the received
request.
The extra "mustAuthenticate" parameter to the Session::sync() method
is replaced with temporarily setting username/password to empty
via the Session::m_syncFilter.
To catch undesired prototype changes of D-Bus API calls (as it
happened when adding that parameter), all these methods now come with
a comment that links them to the corresponding D-Bus API call.
test-dbus.py tests cover the different credential checks:
- TestConnection.testStartSync: invalid MD5 creds, no authorization
necessary => client accepted
- TestConnection.testCredentialsWrong: invalid MD5 creds, authorization
necessary => client rejected
- TestConnection.testCredentialsRight: correct basic creds, authorization
necessary => client accepted
The server was always asking clients to authenticate with
the more secure MD5+Nonce method. For clients which don't
support that, there was no way to tell the server to accept
them.
Changed the configuration so that basic authentication is
accepted. The server will still send a Nonce and ask for MD5 for
failed logins, so clients can (and should) use MD5+Nonce.
The motivation for this change was not a specific client, but
testing (will be committed later): sending valid credentials
in a preformatted message is easier with basic authentication,
exactly because it is susceptible to replay attacks.
Lukas pointed towards <logininitscript>return TRUE</logininitscript>
as a better way of accepting all kinds of sessions, even those with
invalid credentials. For good measure, requestedauth/requiredauth
are still set to none, even though that shouldn't matter now.
The server now checks whether it has other sessions active or in the
queue with the same device ID and kills the older sessions when adding
a new one. Because it does this on each Process(), it is unlikely
(impossible?!) to to have more than one older session, but because the
session might be running or still inactive in the queue, both are
checked.
As part of testing with test-dbus.py it was observed that a connection
that was to be aborted was also closed normally first. That was because
the sync engine did not set an error status in this case. Better check
for abort requests explicitly before calling agent->shutdown(). However,
this should be fixed as part of the server progress handling (MB#7709).
This patch fixes signal handling and shutdown of syncevo-dbus-server
when signals were received. This problems were found in combination
with automated tests via test-dbus.py.
Signals handled by the signal handlers in the core had no effect on
syncevo-dbus-server, which called its own event loop once the
suspended or aborted sync returned. Signals received while outside
of a sync and outside of an event loop also had no effect.
Now signals are always caught by the niam() function in
syncevo-dbus-server. It uses the new SyncContext::handleSignal() API
to tell the core engine about it and in addition, remembers that
syncevo-dbus-server is meant to shut down (suspendRequested). This is
then checked by the syncevo-dbus-server event loop (DBusServer::run())
before blocking again.
The implementation of Session.Abort() and Session.Suspend() didn't
work. isSuspend() and isAbort() mixed up the state. They also did not
wake up DBusTransportAgent, so the syncevo-dbus-server might get stuck
although an abort was already requested. Partially fixed by adding
g_main_loop_quit(). It's not entirely sure whether g_main_loop_quit()
is reentrant (see below). There is other code which uses it in signal
handlers, but that doesn't mean that this is correct. The right long
term solution would be:
- avoid entering and leaving the event loop
- feed signals into the event loop as normal events
SoupTransportAgent solves this by polling for a signal once per second.
This also should be improved.
Because such code was used in several places for a connection, that common
code was moved into Connection::wakeupSession().
Things which went in unnoticed when the original signal handling was merged:
- SyncContext::getSuspendFlags() should not allow the caller to modify
the SyncContext state. Made the returned SuspendFlags const.
- A signal handler may only call re-entrant functions (Stevens, "Advanced
Programming in a Unix Environment", 10.6).
All printing therefore has to be done outside of the signal handlers.
Now the signal handlers just set a message and the regular code
checking for abort/suspend prints it. There's a slight race condition
(a message might get overwritten before it is printed), but printing
just the last message might actually make more sense. There might be
a slight delay between receiving the signal and printing now.
Follow up the previous behavior of syncevolution core
about CTRL-C/SIGINT handling.
Also add SIGTERM handler to abort sync immediately.
currently dbus server checks abort/suspend requests
not only from client but also from 'CTRL-C'/SIGINT
signals.
When the syncevo-dbus-server receives a SyncML message as initial
data from a transport stub, it peeks into the data with the help
of the Synthesis server engine and our SyncEvolution_Session_CheckDevice()
and extracts the LocURI = device ID of the client. This value is
guaranteed to be in the initial SyncML message.
Then it searches for a peer configuration (still called "server
configuration" in the current code) which has the same ID in its
remoteDeviceID property and uses that configuration.
Instantiating SyncContext and Synthesis engine multiple times
is a bit tricky. The context for the SynthesisDBPlugin currently
has to be passed via a global variable, SyncContext::m_activeContext.
SyncContext::analyzeSyncMLMessage() temporarily replaces that
with the help of a new sentinal class SwapContext, which ensures
that the previous context is restored when leaving the function.
Some common code was moved around for this (SyncContext::initEngine()).
The "default config" parameter in syncevo-http-server.py was
removed because it is no longer needed. The possibility to
select some kind of config context via the path below the
sync URL still exists. This is passed to syncevo-dbus-server via
the "config" peer attribute. It has no effect there at the
moment.
TestConnection.testStartSync() in test-dbus.py covers this kind of
config selection. To run it, a peer configuration with remoteDeviceID
"sc-api-nat" must exist.
The "mustAuthenticate" parameter in the D-Bus Server.Connect()
call was ignored, valid credentials were always required unless
username and password in the server configuration were empty.
Now this parameter is passed through and used
- to configure the Synthesis engine correctly
- to suppress asking for our password if not needed
Note that we rely on a patch that turns <requiredauth>none</requiredauth>
into "accept invalid credentials". Without that patch, the client would
have to provide no credentials at all (which SyncEvolution doesn't support).
Starting with this commit, SyncEvolution implements the
Synthesis session authentication and device administration
itself. The goal is to have better control over these steps
and become independent of the SDK_textdb which implemented
this before.
The SDK_textdb works, but it has several limitations, among
them the inability to choose a password. It is also called
"demo" in the source code, which raises some doubts about
its stability.
Because the Synthesis DB interface is now used both for database
access and for session handling, the Module_CreateContext() function
can not always select a source as context. When used as session
plugin, the global context pointer is NULL.
The Synthesis engine calls us at various points to save information.
Because it is not exactly clear how long we can buffer this
information in memory, it is always flushed to disk immediately.
A zero retry interval was accepted and had the effect that
each message send was aborted immediately. This clearly is
not a useful configuration, so now a value of zero is interpreted
as "don't resend".
Also clarified that interval and duration are specified in
seconds.
Previously, sync failed because the datastore configuration specified
no means of storing the admin data (single chunk of text and local ID/remote ID
mapping). This could have been added by configuring the SDK_textdb as
<plugin_admin_module>.
Instead this patch implements the functionality inside SyncEvolution,
using a new ".server.ini" config node for the mapping and an
internal "adminData" property for the admin data text chunk. This is
more natural because it keeps the data under our control.
Authentication is configured via <simpleauthuser/pw>. In contrast to
using the SDK_textdb, this allows choosing the username and password.
Ultimately this should also be done inside SyncEvolution, to avoid
writing the username/password into the XML (unsolved encoding issue)
and into log files (privacy issue).
This uses the new combined client/server Synthesis engine. When
building shared modules, the engine is opened dynamically only
while needed, thus reducing overall memory consumption.
The HTTP server is implemented in Python, using the the 'twisted'
server framework because it can use the same glib event loop as the
D-Bus binding. This allows us to keep the same event loop running
forever and react to both kinds of events.
The server takes a base url (including host name and port)
and a default configuration as name. The host name is currently
ignored. It could be used to bind to a specific interface.
The path is what a client has to use as part of his sync URL
to synchronize against the default configuration. In addition
the client can add another path to select that as his server
configuration.
For example, if the script is called with
http://localhost:9000/syncevolution default
then syncURL = http://localhost:9000/syncevolution will synchronize
against the configuration called "default". With syncURL =
http://localhost:9000/syncevolution/my_server, it will
synchronize against "my_server".
Sources which aren't active during a sync exist in the SyncEvolution
and therefore GUIs may show them. The Synthesis engine isn't told
about these sources, so it won't generate any events for them.
This patch generates the normal 100% done events and the final
source is "done" events inside the EvolutionSyncClient before even
invoking the Synthesis engine. That way GUIs don't have to treat
these inactive sources in any special way.
The command line output is aware of this change and suppresses
output for inactive sources, except for a single INFO line which
tells the user that the source is inactive.
Because of the way the internal API works, it is necessary to
instantiate a dummy SyncSource with no config nodes.
The engine doesn't handle being told twice that data was sent.
SyncEvolution had a code path were this could happen (TransportAgent::wait()
returns without response available). With the existing transports
this never happened, but with the new D-Bus transport it does.
This patch now uses stepCmd = STEPCMD_NEEDDATA for the situation that
SessionStep() doesn't have to be called, thus looping back into the
transport agent directly after checking for abort/suspend.
This TransportAgent API revision was done in preparation for transport
agents which do not support HTTP. Forcing them to provide HTTP specific
methods is unnecessary, so now setting things like proxy is done when
constructing an HTTP-based agent in the createTransport() call.
So now the result of createTransport() must be ready for sending messages.
setURL() is still part of the API, because it provides message-specific
information. Perhaps it should be renamed when it is clearer what
the corresponding information in other transports is.
For connection oriented agents a new shutdown() call was introduced.
This gives them a chance to close the connection and inform the
engine about errors during that shutdown.
For servers (which send the last message without expecting a reply),
wait(noReply=true) was added.
Selecting active sources during a sync or status check was done with a
combination of setting a sync mode via a source config filter and
setting a list of active sources. Now the SyncConfig supports a source
filter which is applied to all sources and source filters for each
source. The latter override the former.
This is powerful enough to start syncs with full control over which
sources are active in which mode, as described in the new D-Bus API.
As part of this patch, the command line semantic is implemented
entirely using a combination of different source filters.
This method complements the corresponding method in SyncSource.
At the moment it does not do more than calling the underlying
Exception::handle(). Calling it via SyncContext
is more natural and might be treated differently in the future.
The intention is to get rid of the historic and inconsistent
naming of some classes and their corresponding files:
* EvolutionSyncClient = class derived from Funambol's SyncClient,
* SyncEvolutionConfig = SyncEvolution's config
With the strict 'namespace SyncEvo' and the syncevo/ path prefix for
most header files it is no longer necessary to have "SyncEvolution" or
"Evolution" in the names. This patch thus renames as follows:
EvolutionSyncClient => SyncContext
EvolutionSmartPtr => SmartPtr
SyncEvolutionCmdline => Cmdline
SyncEvolutionConfig => SyncConfig
SyncEvolutionUtil => util
The former EvolutionSyncClient always had a role that went beyond just
running a sync, for example it also provided config access. With the
upcoming server support it also won't be just a client. Thus the new
name "SyncContext".
The 'syncevo/' prefix is used throughout the code now.
removed whenever the prefix made it clear that the file belongs
to SyncEvolution. This helps finding incorrect include paths.
Quotes should be used exclusively for SyncEvolution files which don't
have a specific prefix yet (test.h, config.h) to help identifying
them.
