recv() may return -1 and EAGAIN/EWOULDBLOCK, according to the man
page. That was treated as a fatal error instead of retrying, which
was the original intention.
For writev() it is less clear whether EAGAIN/EWOULDBLOCK need to be
handled. Let's catch them anyway.
The following code crashed if m_length was read after executing
the release() calls:
realloc(buffer.m_message.release(), buffer.m_message->m_length)
That's because release() resets the m_message pointer. It seems that
clang orders execution like that whereas gcc doesn't.
The main bug fix is for partial writes: the base address wasn't getting
updated, so the same data would have been sent multiple times. Not sure
whether this really occurred in practice.
fcntl() was called for random file descriptors due to a buffer
overflow. If a valid file descriptor was hit, the real user of it
might have gotten confused.
writev() was passed unitialized memory. Okay in this case (hole in struct),
but ugly in valgrind and easily fixed with memset().
Exceptions encountered in the child process were printed as [ERROR],
but not reported to the parent process and thus did not end up in the
final sync report.
Now they are, which makes finding the root cause of a failed local
sync a bit easier.
The error message "@<context>/<source>: source not configured" becames
confusing when <source> is empty. Now "missing URI for one of the
sources" is printed instead.
The methods related to sync username/password were just called
get/setUsername/Password(), with additional check/savePassword()
methods. The savePassword() method clashed with a semantically
different method for arbitrary passwords (as pointed out by clang
2.9), so let's make it more clear that one set of methods specifically
is for the sync username/password.
When the glib event loop is left because the D-Bus client has
requested an abort, the LocalTransportAgent should simply return a
"failed" status and let the caller handle the abort. The return code
of write/readMessage() must be able to convey that - extended from
boolean to an enum.
SyncContext did not do that correctly in server mode: the check for
abort must be done before giving up by throwing an exception.
The D-Bus test now checks that the right status is recorded (wasn't
the case earlier).
Local sync inside syncevo-dbus-server must keep the main event
loop running, otherwise the D-Bus server would stop responding
to D-Bus calls while a sync runs.
This is tested as part of new test-dbus.py tests. They set up local
sync between two directories and then test syncing, timeout detection
when using the glib support code, and aborting via the D-Bus API
while a sync runs.
This testing depends on delaying the sync inside the child
process. Setting the SYNCEVOLUTION_LOCAL_CHILD_DELAY env variable
achieves that (might also be useful for interactive debugging). It is
set for those tests which need it via additional function properties,
similar to the older @timeout() mechanism, which now uses the
more general @property function decorator.
Timeouts for the SyncML message are detected on the server side.
No resending is possible, nor needed: this is a reliable local
transport after all, not HTTP.
The implementation uses select() and non-blocking sockets to time out
at the right time. While at it, it also sets CLOEXEC to ensure that
the pipes are not accidentally inherited by other child programs.
Instead of allowing users of the API to register a callback
which can choose between aborting and time out, only accept
the timeout duration and always treat that as a timeout.
The advanced functionality wasn't used and the simplification
makes implementing the API easier.
Now the @context or @peer@context suffix in sync and source properties
on the command line can be used to configure both sides of the sync
differently.
Source names already contained the context in local sync, both in
parent and child process. But normal messages often did not
distinguish between parent and child, unless the message itself
explicitly remembered to include that information.
Now the child process puts its context into the [] prefix, like this:
[INFO @foobar] xyz...
For some messages this duplicates information, but better too often
than not often enough.
When running client-test, the normal stdout text is copied into
a .log text file named after the currently running test. The output
of the child process should also go there.
This patch achieves that by being more selective about which loggers
in the stack of loggers it removes: LogDir must be removed because it
writes per-process Synthesis .html files, LoggerStdout must remain for
.log. The difference is abstracted away behind a new
Logger::isProcessSafe() method, which tells LocalTransportAgent whether
sharing the logger instance between processes is okay.
The code which resized the buffer for inter-process communication
forgot to update the buffer size, leading to a recv() with zero
size, leading to an error ("Operation not supported") and an aborted
sync.
If an error occurred while receiving the client report, that error
would trigger another attempt to receive the report, and so on, until
the stack was exhausted.
Now the m_statusFD is cleared before the only attempt to get the
report.
The SyncReport is transmitted via an additional socket pair. This is
necessary because the one for messages is shut down to notify the peer
of a premature shutdown.
The parent uses the information obtained from the SyncReport to update
its own status: first the error is logged (thus also setting the
"first error seen" part of the parent), then an exception with a bad child
status is thrown (interrupting anything the parent was doing and, if it
was the first error, setting the overall status of the sync).
There were some issues:
- not calling waitpid() caused zombies if the parent
was a long-running process
- not closing the socket on the parent side caused
the child to get stuck in error cases
- exit() has undesired side effects on the parent,
use _exit() to just terminate the child without
additional cleanup (more like fork/exec that way)
This patch also adds some debug output for tracking down
such issues.
Temporarily setting --sync-property values only affects the server
side of local sync. This patch extends a hack which copies some
relevant values over into the client:
- do it sooner
- copy additional properties
A better way to set these properties on both sides of local sync is
needed.
In local sync the terms "local" and "remote" (in SyncReport, "Data
modified locally") do not always apply and can be confusing. Replaced
with explicitly mentioning the context.
The source name also no longer is unique. Extended in the local sync
case (and only in that case) by adding a <context>/ prefix to the
source name.
Here is an example of the modified output:
$ syncevolution google
[INFO] @default/itodo20: inactive
[INFO] @default/addressbook: inactive
[INFO] @default/calendar+todo: inactive
[INFO] @default/memo: inactive
[INFO] @default/ical20: inactive
[INFO] @default/todo: inactive
[INFO] @default/file_calendar+todo: inactive
[INFO] @default/file_vcard21: inactive
[INFO] @default/vcard30: inactive
[INFO] @default/text: inactive
[INFO] @default/file_itodo20: inactive
[INFO] @default/vcard21: inactive
[INFO] @default/file_ical20: inactive
[INFO] @default/file_vcard30: inactive
[INFO] @google/addressbook: inactive
[INFO] @google/memo: inactive
[INFO] @google/todo: inactive
[INFO] @google/calendar: starting normal sync, two-way
Local data changes to be applied remotely during synchronization:
*** @google/calendar ***
after last sync | current data
removed since last sync <
> added since last sync
-------------------------------------------------------------------------------
BEGIN:VCALENDAR BEGIN:VCALENDAR
...
END:VCALENDAR END:VCALENDAR
-------------------------------------------------------------------------------
[INFO] @google/calendar: sent 1/2
[INFO] @google/calendar: sent 2/2
Local data changes to be applied remotely during synchronization:
*** @default/calendar ***
no changes
[INFO] @default/calendar: started
[INFO] @default/calendar: updating "created in Google, online"
[INFO] @default/calendar: updating "created in Google - mod2, online"
[INFO] @google/calendar: started
[INFO] @default/calendar: inactive
[INFO] @google/calendar: normal sync done successfully
Synchronization successful.
Changes applied during synchronization:
+---------------|-----------------------|-----------------------|-CON-+
| | @default | @google | FLI |
| Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| calendar | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| disabled, 0 KB sent by client, 2 KB received |
| item(s) in database backup: 3 before sync, 3 after it |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| start Mon Oct 25 10:03:24 2010, duration 0:13min |
| synchronization completed successfully |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
Data modified @default during synchronization:
*** @default/calendar ***
before sync | after sync
removed during sync <
> added during sync
-------------------------------------------------------------------------------
BEGIN:VCALENDAR BEGIN:VCALENDAR
VERSION:2.0 VERSION:2.0
...
END:VCALENDAR END:VCALENDAR
-------------------------------------------------------------------------------
pohly@pohly-mobl1:/tmp/syncevolution/src$
Synchronization successful.
Changes applied during synchronization:
+---------------|-----------------------|-----------------------|-CON-+
| | @google | @default | FLI |
| Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| calendar | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0 | 0 |
| two-way, 2 KB sent by client, 0 KB received |
| item(s) in database backup: 2 before sync, 2 after it |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
| start Mon Oct 25 10:03:24 2010, duration 0:13min |
| synchronization completed successfully |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
Data modified @google during synchronization:
*** @google/calendar ***
no changes
During local sync names like "addressbook" are no longer unique,
because they may exist in both the local and the remote context. This
patch introduces a "display name" composed from context and source name,
like this: "@<context>/<source>" (@default/addressbook).
The context is only used if needed, which currently is the case during
a local sync.
Changing the SyncSourceBase::getName() result was also considered, but
several places expect this to be the name of the source inside its
context, so an explicit SyncSourceBase::getDisplayName() turned out
to be safer.
Local sync with a "source-config" peer config for the sources
did not quite work right yet. This patch fixes several issues:
- double @ sign in source-config@<context>
- sync = disabled must be set explicitly before checking
activated sources, because source-config@<context> may
have some of these sources enabled (depends on persistent
SyncSourceNodes)
- overriding the sync URL is not needed (transport agent already
set explicitly) and harmful (original value needed by sources)
Backends like XMLRPC need information about URL, proxy and SSL
settings, etc. This can be done via source specific properties, like
evolutionsource, but this is not how this is normally done. It would
be nicer if the exising per-peer properties could be used. The goal is
that a normal peer configuration can be created from a template with
the necessary information to enable sources using that information.
This patch makes this possible by adding a context parameter to
SyncSourceParams:
* @param context Additional non-source config settings.
* When running as part of a normal sync, these are the
* settings for the peer. When running in a local sync,
* these settings come from the "source-config" peer
* config inside the config context of the source.
* Testing uses "source-config@client-test". On the
* command line, this is the config chosen by the
* user, which may or may not have peer-specific settings!
Note that this still doesn't solve the problem for XMLRPC to SyncML
peer sync, because in that case ("normal sync") the context will be
the one describing the peer. SyncURL is already used and proxy
settings might not match.
The XMLRPC backends therefore was not changed and continues to use
evolutionsource.
Don't allow sync inside same context, too easy to configure incorrectly.
Also don't allow referencing a peer (leads to incorrect session dirs
in client and cannot override peer files).
Local sync is configured with a new syncURL = local://<context> where
<context> identifies the set of databases to synchronize with. The
URI of each source in the config identifies the source in that context
to synchronize with.
The databases in that context run a SyncML session as client. The
config itself is for a server. Reversing these roles is possible by
putting the config into the other context.
A sync is started by the server side, via the new LocalTransportAgent.
That agent forks, sets up the client side, then passes messages
back and forth via stream sockets. Stream sockets are useful because
unexpected peer shutdown can be detected.
Running the server side requires a few changes:
- do not send a SAN message, the client will start the
message exchange based on the config
- wait for that message before doing anything
The client side is more difficult:
- Per-peer config nodes do not exist in the target context.
They are stored in a hidden .<context> directory inside
the server config tree. This depends on the new "registering nodes
in the tree" feature. All nodes are hidden, because users
are not meant to edit any of them. Their name is intentionally
chosen like traditional nodes so that removing the config
also removes the new files.
- All relevant per-peer properties must be copied from the server
config (log level, printing changes, ...); they cannot be set
differently.
Because two separate SyncML sessions are used, we end up with
two normal session directories and log files.
The implementation is not complete yet:
- no glib support, so cannot be used in syncevo-dbus-server
- no support for CTRL-C and abort
- no interactive password entry for target sources
- unexpected slow syncs are detected on the client side, but
not reported properly on the server side
Patrick Ohly