SyncEvolution 1.5.3 -> 2.0.0, 21.03.2021 ======================================== This release modernizes the code base and removes usage of out-dated libraries and APIs. All Python scripts require Python 3. The major version bump reflects that this release is not just a minor bug fix. However, no new features were added. Binaries on syncevolution.org get built for distros >= Ubuntu Bionic and Debian Buster. Testing is now based on Docker containers instead of a custom schroot solution, so adding testing against other distros will be easier in the future. Compilation on Fedora Rawhide was already added. Some features are no longer built and thus untested: - ActiveSync - KDE The code is still there, but needs help from interested developers to ensure that it really works. It may get removed in a future version if there is no interest. SyncEvolution 1.5.2 -> 1.5.3, 03.01.2018 ======================================== Maintenance release. syncevolution.org binaries are now getting compiled for distros >= Ubuntu Xenial 16.04 LTS. Usage of deprecated libraries (GNOME keyring) and APIs (SoupAsyncSession) was replaced. libical v3 is supported. The code now compiles more cleanly with recent compilers and depends on C++11 support. Details: * EDS: more generic open retry handling Recent EDS started to exhibit race conditions when opening a database (for example, https://bugzilla.gnome.org/show_bug.cgi?id=791306). Opening was already tried again for a certain known error in some old EDS version. Now it is tried five times with a delay of one second for all errors. * SoupTransportAgent: require libsoup 2.42, no deprecated methods This allows us to get rid of deprecated function calls. We no longer need to set a default proxy either, the newer libsoup does that itself by default. * C++: replace auto_ptr with unique_ptr, require C++11 auto_ptr has been deprecated for a while now. unique_ptr can be taken for granted now, so use that instead. * testing: work around Google CalDAV RECURRENCE-ID Stand-alone events with RECURRENCE-ID get mangled by the server: it converts the RECURRENCE-ID time to UTC. Reported in: https://stackoverflow.com/questions/47811670/detached-recurrence-without-parent-event * GNOME: replace gnome-keyring with libsecret (FDO #104219) The GNOME keyring library has been obsoleted for a long time now, long enough that the replacement libsecret is available on all supported distros. Therefore we can switch unconditionally. * libical: support libical v3 (FDO #104220) libical v3 removes some deprecated functions (like icaltime_from_timet) and removes the "is_utc" member from icaltimetype. The replacement code works with old and new libical and thus needs no ifdefs. Original author: Milan Crha * syncevolution.org: fixed packaging (FDO #98014, FDO #100549) The activesyncd package missing dependencies on libgnome-keyring0 and libglib2.0-bin and therefore failed to work when installed on a minimal system without those. * various build and test fixes/workarounds SyncEvolution 1.5.1 -> 1.5.2, 08.11.2016 ======================================== Maintenance release. syncevolution.org binaries are now getting compiled for distros >= Ubuntu Trusty 14.04 LTS, which allowed removing several hacks that were needed when building binaries that also had to run on older distros. Compilation from source for old distros should still work as before, but is not getting tested anymore. Compile problems with recent libraries (libical v2) and tools (GCC v6) were resolved. Syncing via Bluetooth with certain phones now should work reliably in incremental mode. New backends for the Trinity Desktop Environment (TDE) were added to the source code. Details: * ObexTransportAgent.cpp: properly shut down connection (FDO #91485) Apparently there's a race condition in the OBEX transport that causes the connection to phones via Bluetooth to be shut down prematurely. Some phones react by doing a slow sync instead of an incremental sync the next time. * support non-readable parent directories (FDO #91000) The previous mkdir_p() walked down top to bottom and checked each path entry as it went along. That approach failed unnecessarily when some existing parent directory could not be read (non-readable /home, for example). * avoid using dbus-launch (Debian #836399) dbus-launch is considered deprecated because of the X11 dependency. See https://lists.debian.org/debian-devel/2016/08/msg00554.html "Mass bug filing: use and misuse of dbus-launch (dbus-x11)" The dbus-session.sh script still needs to start the D-Bus daemon when used in the nightly testing, so the code now does it by invoking the dbus-daemon directly. syncevo-http-server still has some usage of dbus-launch left, but that's strictly for systems which don't have the more modern D-Bus. * syncevo-dbus-server integrates better with systemd (FDO #92164) A .service file allows the D-Bus daemon to start the service via systemd, thus ensuring that the process environment is correct. Patch from Simon McVittie. Auto-starting as part of the desktop login uses D-Bus activation if the "dbus-send" tool is installed. * syncevolution.org: compile on Ubuntu Trusty, libical v1/v2 compatibility syncevolution.org binaries are now getting compiled on Ubuntu Trusty and thus no longer support distros with older EDS. The code should still compile against older EDS (for example, for Maemo), but that is not getting tested anymore. This allows removing the dynamic linker hacks related to older libraries, which was only used in those binaries. Instead, backends using libical or EDS get compiled on Ubuntu Trusty and then the soname of those libs get patched to make the backend module usable in combination with a different set of libs. That patching is part of a script maintained in the syncevolution.org build infrastructure. This approach was already used before to generate different EDS backends for EDS versions with the newer EClient API, because that turned out to be easier than the dynamic loading approach. It works because none of the methods used by SyncEvolution changed their ABI, only some other parts of the libraries did. Should there ever be a situation again that cannot be handled like this, then backends also get compiled on different distros than Ubuntu Trusty (for example, the ActiveSync backend for Debian Stretch is built on Debian Stretch). libical still requires one special hack: system time zone loading in libical v1 (and only in that version, v2 has builtin support again) must be overridden such that time zones are generated with rules instead of transitions because that is more compatible with the peers that SyncEvolution exchanges data with. That hack now relies on overriding the two relevant functions inside the main binaries (has to be there, otherwise libical still ends up calling its own internal implementation). The overriding code is in libsyncevo-icaltz-util.so.0 and depends on libical.so.1. If libsyncevo-icaltz-util.so.0 can be loaded, the wrappers in the main binary use it, otherwise they fall through to the code from the current libical.so, which then should be libical.so.2 or more recent. This hack is active by default when libical v1 is detected during configuration. * optionally show debug output in --version output SYNCEVOLUTION_DEBUG=1 syncevolution --daemon=no --version now dumps also the debug information gathered by the binary compatibility code. It was only available in sync logs before. * various build fixes for libical v2, GCC v6/C++14 SyncEvolution 1.5 -> 1.5.1, 05.06.2015 ====================================== Maintenance release. Binaries now also get compiled for Debian 8.0 "Jessie". Details: * avoid time zone issue with Funambol server The Funambol iCalendar 2.0 parser fails to handle time zones with quotation marks around the TZID value, which is something that SyncEvolution started to add in 1.4.99.3. While it is valid to quote like that, it is not necessary, so avoid quoting in this case to restore interoperability. * syncevo-http-server: stop using deprecated twisted.web.error (FDO #90419) This has become a real problem for example on Fedora 22 where the old name is no longer available. * syncevo-http-server: use TLS instead of SSLv3 This fixes a potential security risk and connection problems with clients that don't support SSLv3 anymore. * syncing: avoid segfault for invalid text inside items (FDO #90118) As reported by Canonical, syncing fails if data items contain text which is not correct UTF-8 in one of the fields that SyncEvolution logs in the command line output (like SUMMARY of a calendar event). That is because the byte string coming from the item is passed unchecked to the D-Bus implementation for transmission via D-Bus. But D-Bus strings must be correct UTF-8, so depending on the D-Bus library in use, one gets a segfault (GIO D-Bus, due to an unchecked NULL pointer access) or an "out of memory" error (libdbus, which checks for NULL). SyncEvolution now replaces invalid bytes with a question mark in its output while preserving the rest of the text. * file backend: log item manipulation Extracting a meaningful description of each item from the Synthesis engine when updating and adding items is easy to do for items of certain known types (contacts and calendar items). * command line: preserve log prefix of target side of local sync In some cases, the prefix which was supposed to be embedded in the log messages from the target side of a local sync got lost on the way to the command line tool. Primarily this affected the added/updated/deleted messages, as in: [INFO remote@client] @client/addressbook: started [INFO remote@client] updating "Joan Doe" [INFO remote@client] @client/addressbook: received 1/1 * compile fix: use ${PKG_CONFIG} instead of pkg-config. This fixes the build on Exherbo that only has prefixed versions of pkg-config. * WebDAV: handle 403 during Google OAuth authentication When sending an access token with insufficient scope (for example, because the Ubuntu Online Accounts service definition was incomplete, as documented in FDO #86824), Google responds with a 403 "service denied" error. This is now dealt with by retrying, just as for a transient 401 error. * CalDAV: more efficient "is empty" check (FDO #86335) Since 1.4.99.4, syncing WebDAV collections always checks first whether there are items in the collections. This was partly done for slow sync prevention (which is not necessary for empty collections), partly for the "is the datastore usable" check. However, this did not take into account that for CalDAV collections, the entire content gets downloaded for this check. That is because filtering by item type (VEVENT vs. VJOURNAL) is not implemented correctly by all servers. So now all CalDAV syncs, whether incremental or slow, always transfered all items, which is not the intention (incremental syncs should be fast and efficient). This release adds a more efficient isEmpty() check: for simple CardDAV collections, only luid and etag get transferred, as in listAllItems(). This is the behavior from 1.5. For CalDAV, a report with a filter for the content type is used and the transfer gets aborted after the first item, without actually double-checking the content of the item. This is different from listAllItems(), which really transfers the content. This extra content check would only be needed for some old servers (Radical 0.7) and is not essential, because reporting "not empty" even when empty is safe. * WebDAV: send Basic Auth via http in some cases (FDO #57248) It turned out that finding databases on an Apple Calendar server accessed via http depends on sending Basic Auth even when the server does not ask for it: without authentication, there is no information about the current principal, which is necessary for finding the user's databases. To make this work again, sending the authentication header is now forced for plain http if (and only if) the request which should have returned the principal URL fails to include it. This implies sending the same request twice, but as this scenario should be rare in practise (was only done for testing), this is acceptable. * Ubuntu Online Accounts: support plain text credentials The backend for UOA was rewritten by Alberto Mardegan and now also can use plain username/password credentials stored in UOA. * various compiler error and warning fixes SyncEvolution 1.4.1 -> 1.5, 31.10.2014 ====================================== Based on community feedback and discussions, the terminology used in SyncEvolution for configuration, local sync and database access was revised. Some usability issues with setting up access to databases were addressed. Interoperability with WebDAV servers and in particular Google Contacts was enhanced considerably. Access to iCloud contacts was reported as working when using username=foobar@icloud.com and password, but is not formally tested. Syncing with iCloud calendars ran into a server limitation (reported as 17001498 "CalDAV REPORT drops calendar data") and needs further work (FDO #72133). Contact data gets converted to and from the format typically used by CardDAV servers, so now anniversary, spouse, manager, assistant and instant message information are exchanged properly. Custom labels get stored in EDS as extensions and no longer get lost when updating some other aspects of a contact. However, Evolution does not show custom labels and removes them when editing a property which has a custom label (BGO #730636). Scanning for CardDAV/CalDAV resources was enhanced. It now finds additional calendars with Google CalDAV. For Google, the obsolete SyncML config template was removed and CalDAV/CardDAV were merged into a single "Google" template. Using Google Calendar/Contacts with OAuth2 authentication on a headless server becomes a bit easier: it is possible to set up access on one system with a GUI using either gSSO or GNOME Online Accounts, then take the OAuth2 refresh token and use it in SyncEvolution on a different system. See http://cgit.freedesktop.org/SyncEvolution/syncevolution/tree/src/backends/oauth2/README The PIM Manager API also supports Google Contact syncing. Some problems with suspending a PBAP sync were fixed. Suspend/abort can be tested with the sync.py example. Performance is better for local syncs and PBAP caching. The most common case, a two-way sync with no changes on either side, no longer rewrites any meta data files. CPU consumption during local sync was reduced to one third by exchanging messages via shared memory instead of internal D-Bus. Redundant vCard decode/encode on the sending side of PBAP and too agressive flushing of meta data during a normal sync were removed. The EDS memo backend is able to switch between syncing in plain text and iCalendar 2.0 VJOURNAL automatically. Details: * source -> datastore rename, improved terminology The word "source" implies reading, while in fact access is read/write. "datastore" avoids that misconception. Writing it in one word emphasizes that it is single entity. While renaming, also remove references to explicit --*-property parameters. The only necessary use today is "--sync-property ?" and "--datastore-property ?". --datastore-property was used instead of the short --store-property because "store" might be mistaken for the verb. It doesn't matter that it is longer because it doesn't get typed often. --source-property must remain valid for backward compatility. As many user-visible instances of "source" as possible got replaced in text strings by the newer term "datastore". Debug messages were left unchanged unless some regex happened to match it. The source code will continue to use the old variable and class names based on "source". Various documentation enhancements: Better explain what local sync is and how it involves two sync configs. "originating config" gets introduces instead of just "sync config". Better explain the relationship between contexts, sync configs, and source configs ("a sync config can use the datastore configs in the same context"). An entire section on config properties in the terminology section. "item" added (Todd Wilson correctly pointed out that it was missing). Less focus on conflict resolution, as suggested by Graham Cobb. Fix examples that became invalid when fixing the password storage/lookup mechanism for GNOME keyring in 1.4. The "command line conventions", "Synchronization beyond SyncML" and "CalDAV and CardDAV" sections were updated. It's possible that the other sections also contain slightly incorrect usage of the terminology or are simply out-dated. * local sync: allow config name in syncURL=local:// Previously, only syncURL=local://@ was allowed and used the "target-config@context name" config as target side in the local sync. Now "local://config-name@context-name" or simply "local://config-name" are also allowed. "target-config" is still the fallback if only a context is give. It also has one more special meaning: "--configure target-config@google" will pick the "Google" template automatically because it knows that the intention is to configure the target side of a local sync. It does not know that when using some other name for the config, in which case the template (if needed) must be specified explicitly. The process name in output from the target side now also includes the configuration name if it is not the default "target-config". * command line: revise usability checking of datastores When configuring a new sync config, the command line checks whether a datastore is usable before enabling it. If no datastores were listed explicitly, only the usable ones get enabled. If unusable datastores were explicitly listed, the entire configure operation fails. This check was based on listing databases, which turned out to be too unspecific for the WebDAV backend: when "database" was set to some URL which is good enough to list databases, but not a database URL itself, the sources where configured with that bad URL. Now a new SyncSource::isUsable() operation is used, which by default just falls back to calling the existing Operations::m_isEmpty. In practice, all sources either check their config in open() or the m_isEmpty operation, so the source is usable if no error is enountered. For WebDAV, the usability check is skipped because it would require contacting a remote server, which is both confusing (why does a local configure operation need the server?) and could fail even for valid configs (server temporarily down). The check was incomplete anyway because listing databases gave a fixed help text response when no credentials were given. For usability checking that should have resulted in "not usable" and didn't. The output during the check was confusing: it always said "listing databases" without giving a reason why that was done. The intention was to give some feedback while a potentially expensive operation ran. Now the isUsable() method itself prints "checking usability" if (and only if!) such a check is really done. Sometimes datastores were checked even when they were about to be configure as "disabled" already. Now checking such datastores is skipped. * command line: fix --update from directory The "--update " operation was supposed to take the item luids from the file names inside the directory. That part had not been implemented, turning the operation accidentally into an "--import". Also missing was the escaping/unescaping of luids. Now the same escaping is done as in command line output and command line parsing to make the luids safe for use as file name. * sync output: hide ": started" INFO messages These messages get printed at the start of processing each SyncML message. This is not particularly useful and just adds noise to the output. * config: allow storing credentials for email address When configuring a WebDAV server with username = email address and no URL (which is possible if the server supports service discovery via the domain in the email address), then storing the credentials in the GNOME keyring used to fail with "cannot store password in GNOME keyring, not enough attributes". That is because GNOME keyring seemed to get confused when a network login has no server name and some extra safeguards were added to SyncEvolution to avoid this. To store the credentials in the case above, the email address now gets split into user and domain part and together get used to look up the password. * config: ignore unnecessary username property A local sync or Bluetooth sync do not need the 'username' property. When it is set despite that, issue a warning. Previously, the value was checked even when not needed, which caused such syncs to fail when set to something other than a plain username. * config templates: Funambol URLs Funambol turned of the URL redirect from my.funambol.com to onemedia.com. The Funambol template now uses the current URL. Users with existing Funambol configs must updated the syncURL property manually to https://onemediahub.com/sync Kudos to Daniel Clement for reporting the change. * EDS: memo syncing as iCalendar 2.0 (FDO #52714) When syncing memos with a peer which also supports iCalendar 2.0 as data format, the engine will now pick iCalendar 2.0 instead of converting to/from plain text. The advantage is that some additional properties like start date and categories can also be synchronized. The code is a lot simpler, too, because the EDS specific iCalendar 2.0 <-> text conversion code can be removed. * SoupTransport: drop CA file check It used to be necessary to specify a CA file for libsoup to enable SSL certificate checking. Nowadays libsoup uses the default CA store unless told otherwise, so the check in SyncEvolution became obsolete. However, now there is a certain risk that no SSL checking is done although the user asked for it (when libsoup is not recent enough or compiled correctly). * CardDAV: use Apple/Google/CardDAV vCard flavor In principle, CardDAV servers support arbitrary vCard 3.0 data. Extensions can be different and need to be preserved. However, when multiple different clients or the server's Web UI interpret the vCards, they need to agree on the semantic of these vCard extensions. In practice, CardDAV was pushed by Apple and Apple clients are probably the most common clients of CardDAV services. When the Google Contacts Web UI creates or edits a contact, Google CardDAV will send that data using the vCard flavor used by Apple. Therefore it makes sense to exchange contacts with *all* CardDAV servers using that format. This format could be made configurable in SyncEvolution on a case-by-case basis; at the moment, it is hard-coded. During syncing, SyncEvolution takes care to translate between the vCard flavor used internally (based on Evolution) and the CardDAV vCard flavor. This mapping includes: X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE Any IMPP property declared as X-SERVICE-TYPE=AIM will get mapped to X-AIM. Same for others. Some IMPP service types have no known X- property extension; they are stored in EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE (for example, GaduGadu and Groupwise) are stored with X-SERVICE-TYPE values chosen by SyncEvolution so that Google CardDAV preserves them (GroupWise with mixed case got translated by Google into Groupwise, so the latter is used). Google always sends an X-ABLabel:Other for IMPP. This is ignored because the service type overrides it. The value itself also gets transformed during the mapping. IMPP uses an URI as value, with a chat protocol (like "aim" or "xmpp") and some protocol specific identifier. For each X- extension the protocol is determined by the property name and the value is the protocol specific identifier without URL encoding. X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel The mapping is based on the X-ABLabel property attached to the X-ABRELATEDNAMES property. This depends on the English words "Spouse", "Manager", "Assistant" that Google CardDAV and Apple devices seem to use regardless of the configured language. As with IMPP, only the subset of related names which have a corresponding X- property extension get mapped. The rest is stored in EDS using the X-ABRELATEDNAMES property. X-ANNIVERSARY <-> X-ABDATE Same here, with X-ABLabel:Anniversary as the special case which gets mapped. X-ABLabel parameter <-> property CardDAV vCards have labels attached to arbitrary other properties (TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags: item1.X-ABDATE:2010-01-01 item1.X-ABLabel:Anniversary The advantage is that property values can contain arbitrary characters, including line breaks and double quotation marks, which is not possible in property parameters. Neither EDS nor KDE (judging from the lack of responses on the KDE-PIM mailing list) support custom labels. SyncEvolution could have used grouping as it is done in CardDAV, but grouping is not used much (not at all?) by the UIs working with the vCards in EDS and KDE. It seemed easier to use a new X-ABLabel parameter. Characters which cannot be stored in a parameter get converted (double space to single space, line break to space, etc.) during syncing. In practice, these characters don't appear in X-ABLabel properties anyway because neither Apple nor Google UIs allow entering them for custom labels. The "Other" label is used by Google even in case where it adds no information. For example, all XMPP properties have an associated X-ABLabel=Other although the Web UI does not provide a means to edit or show such a label. Editing the text before the value in the UI changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for other fields. Therefore the "Other" label is ignored by removing it during syncing. X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the order of properties in the UI) gets stored in CardDAV. The only exception is Google CardDAV which got confused when an IMPP property had both X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google, X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering of chat information can get lost when syncing with Google. * synccompare: support grouping and quoted parameter strings Grouped properties are sorted first according to the actual property name, then related properties are moved to the place where their group tag appears first. The first grouped property gets a "- " prefix, all following ones are just indended with " ". The actual group tag is not part of the normalized output, because its value is irrelevant: BDAY:19701230 - EMAIL:john@custom.com X-ABLabel:custom-label2 ... FN:Mr. John 1 Doe Sr. - IMPP;X-SERVICE-TYPE=AIM:aim:aim X-ABLabel:Other ... - X-ABDATE:19710101 X-ABLabel:Anniversary Redundant tags (those set for only a single property, X-ABLabel:Other) get removed as part of normalizing an item. * WebDAV: use server's order when listing collections When doing a recursive scan of the home set, preserve the order of entries as reported by the server and check the first one first. The server knows better which entries are more relevant for the user (and thus should be the default) or may have some other relevant order. Previously, SyncEvolution replaced that order with sorting by URL, which led to a predictable, but rather meaningless order. For example, Google lists the users own calendar first, followed by the shared calendars sorted alphabetical by their name. Now SyncEvolution picks the main calendar as default correctly when scanning from https://www.google.com/calendar/dav/. * WebDAV: improved database search (Google, Zimbra) Zimbra has a principal URL that also serves as home set. When using it as start URL, SyncEvolution only looked the URL once, without listing its content, and thus did not find the databases. When following the Zimbra principal URL indirectly, SyncEvolution did check all of the collections there recursively. Unfortunately that also includes many mail folders, causing the scan to abort after checking 1000 collections (an internal safe guard). The solution for both includes tracking what to do with a URL. For the initial URL, only meta data about the URL itself gets checked. Recursive scanning is only done for the home set. If that home set contains many collections, scanning is still slow and may run into the internal safe guard limit. This cannot be avoided because the CalDAV spec explicitly states that the home set may contain normal collections which contain other collections, so a client has to do the recursive scan. When looking at a specific calendar, Google CalDAV does not report what the current principal or the home set is and therefore SyncEvolution stopped after finding just the initial calendar. Now it detects the lack of meta information and adds all parents also as candidates that need to be looked at. The downside of this is that it doesn't know anything about which parents are relevant, so it ends up checking https://www.google.com/calendar/ and https://www.google.com/. In both cases Basic Auth gets rejected with a temporary redirect to the Google login page, which is something that SyncEvolution must ignore immediately during scanning without applying the resend workaround for "temporary rejection of valid credentials" that can happen for valid Google CalDAV URLs. * WebDAV: enhanced database search (Google Calendar) Additional databases where not found for several reasons. SyncEvolution ignored all shared calendars (http://calendarserver.org/ns/shared) and Google marks the additional calendars that way. The other problem was that the check for leaf collections (= collections which cannot contain other desired collections) incorrectly excluded those collections instead of only preventing listing of their content. With this change, https://www.google.com/calendar/dav/?SyncEvolution=Google can be used as starting point for Google Calendar. * WebDAV: fix database scan on iCloud The calendar home set URL on iCloud (the one ending in /calendars/) is declared as containing calendar data. That was enough for SyncEvolution to accept it incorrectly as calendar. However, the home set only contains calendar data indirectly. * WebDAV: support redirects between hosts and DNS SRV lookup based on URL When finding a new URL, we must be prepared to reinitialize the Neon session with the new host settings. iCloud does not have .well-known support on its www.icloud.com server. To support lookup with a non-icloudd.com email address, we must do DNS SRV lookup when access to .well-known URLs fails. We do this without a www prefix on the host first, because that is what happens to work for icloud.com. With these changes it becomes possible to do database scans on Apple iCloud, using syncURL=https://www.icloud.com or syncURL=https://icloud.com. Giving the syncURL like this is only necessary for a username that does not end in @icloud.com. When the syncURL is not set, the domain for DNS SRV lookup is taken from the username. * WebDAV: more efficient item creation PUT has the disadvantage that a client needs to choose a name and then figure out what the real name on the server is. With Google CardDAV that requires sending another request and only works because the server happens to remember the original name (which is not guaranteed!). POST works for new items without a name and happens to be implemented by Google such that the response already includes all required information (new name and revision string). POST is checked for as described in RFC 5995 once before creating a new item. Servers which don't support it continue to get a PUT. * WebDAV: send "User-Agent: SyncEvolution" Apple iCloud servers reject requests unless they contain a User-Agent header. The exact value doesn't seem to matter. Making the string configurable might be better, but can still be done later when it is more certain whether and for what it is needed. * WebDAV: refactor and fix DNS SRV lookup The syncevo-webdav-lookup script was not packaged. It did not report "not found" DNS results correctly and the caller did not check for this either, so when looking up the information for a domain which does not have DNS SRV entries, SyncEvolution ended up retrying for while as if there had been a temporary lookup problem. * Google: remove SyncML template, combine CalDAV/CardDAV Google has turned off their SyncML server, so the corresponding "Google Contacts" template became useless and needs to be removed. It gets replaced by a "Google" template which combines the three different URLs currently used by Google for CalDAV/CardDAV. This new template can be used to configure a "target-config@google" with default calendar and address book database already enabled. The actual URL of these databases will be determined during the first sync using them. The template relies on the WebDAV backend's new capability to search multiple different entries in the syncURL property for databases. To avoid listing each calendar twice (once for the legacy URL, once with the new one) when using basic username/password authentication, the backend needs a special case for Google and detect that the legacy URL does not need to be checked. * Google Calendar: remove child hack, improve alarm hack (FDO #63881) Google recently enhanced support for RECURRENCE-ID, so SyncEvolution no longer needs to replace the property when uploading a single detached event with RECURRENCE-ID. However, several things are still broken in the server, with no workaround in SyncEvolution: - Removing individual events gets ignored by the server; a full "wipe out server data" might work (untested). - When updating the parent event, all child events also get updated even though they were included unchanged in the data sent by SyncEvolution. - The RECURRENCE-ID of a child event of an all-day recurring event does not get stored properly. - The update hack seems to fail for complex meetings: uploading them once and then deleting them seems to make uploading them again impossible. All of these issues were reported to Google and are worked on there, so perhaps the situation will improve. In the meantime, syncing with Google CalDAV should better be limited to: - Downloading a Google calendar in one-way mode. - Two-way syncing of simple calendars without complex meeting serieses. While updating the Google workarounds, the alarm hack (sending a new event without alarms twice to avoid the automatic server side alarm) was simplified. Now the new event gets sent only once with a pseudo-alarm. * CardDAV: implement read-ahead Instead of downloading contacts one-by-one with GET, SyncEvolution now looks at contacts that are most likely going to be needed soon and gets all of them at once with addressbook-multiget REPORT. The number of contacts per REPORT is 50 by default, configurable by setting the SYNCEVOLUTION_CARDDAV_BATCH_SIZE env variable. This has two advantages: - It avoids round-trips to the server and thus speeds up a large download (100 small contacts with individual GETs took 28s on a fast connection, 3s with two REPORTs). - It reduces the overall number of requests. Google CardDAV is known to start issuing intermittent 401 authentication errors when the number of contacts retrieved via GET is too large. Perhaps this can be avoided with addressbook-multiget. * oauth2: new backend using libsoup/libcurl New backend implements identity provider for obtaining OAuth2 access token for systems without HMI support. Access token is obtained by making direct HTTP request to OAuth2 server and using refresh token obtained by user in some other way. New provider automatically updates stored refresh token when OAuth2 server is issuing new one. * signon: make Accounts optional The new "signon" provider only depends on lib[g]signon-glib. It uses gSSO if found, else UOA. Instead of pulling parameters and the identity via libaccounts-glib, the user of SyncEvolution now has to ensure that the identity exists and pass all relevant parameters in the "signon:" username. * gSSO: adapt to gSSO >= 2.0 * signon: fix build Static build was broken for gSSO and UOA (wrong path name to .la file) and gSSO was not enabled properly (wrong condition check). * datatypes: raw text items with minimal conversion (FDO #52791) When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution "databaseFormat", all parsing and conversion is skipped. The backend's data is identical to the item data in the engine. Finding duplicates in a slow sync is very limited when using these types because the entire item data must match exactly. This is useful for the file backend when the goal is to store an exact copy of what a peer has or for limited, read-only backends (PBAP). The downside of using the raw types is that the peer is not given accurate information about which vCard or iCalendar properties are supported, which may cause some peers to not send all data. * datatypes: text/calendar+plain revised heuristic When sending a VEVENT, DESCRIPTION was set to the SUMMARY if empty. This may have been necessary for some peers, but for notes (= VJOURNAL) we don't know that (hasn't been used in the past) and don't want to alter the item unnecessarily, so skip that part and allow empty DESCRIPTION. When receiving a plain text note, the "text/calendar+plain" type used to store the first line as summary and the rest as description. This may be correct in some cases and wrong in others. The EDS backend implemented a different heuristic: there the first line is copied into the summary and stays in the description. This makes a bit more sense (the description alone is always enough to understand the note). Therefore and to avoid behavioral changes for EDS users when switching the EDS backend to use text/calendar+plain, the engine now uses the same approach. * datatypes: avoid PHOTO corruption during merge (FDO #77065) When handling an update/update conflict (both sides of the sync have an updated contact) and photo data was moved into a local file by EDS, the engine merged the file path and the photo data together and thus corrupted the photo. The engine does not know about the special role of the photo property. This needs to be handled by the merge script, and that script did not cover this particular situation. Now the loosing side is cleared, causing the engine to then copy the winning side over into the loosing one. Found by Renato Filho/Canonical when testing SyncEvolution for Ubuntu 14.04. * vcard profile: avoid data loss during merging When resolving a merge conflict, repeating properties were taken wholesale from the winning side (for example, all email addresses). If a new email address had been added on the loosing side, it got lost. Arguably it is better to preserve as much data as possible during a conflict. SyncEvolution now does that in a merge script by checking which properties in the loosing side do not exist in the winning side and copying those entries. Typically only the main value (email address, phone number) is checked and not the additional meta data (like the type). Otherwise minor differences (for example, both sides have same email address, but with different types) would lead to duplicates. Only addresses are treated differently: for them all attributes (street, country, city, etc.) are compared, because there is no single main value. * engine: UID support in contact data Before, the UID property in a vCard was ignored by the engine. Backends were responsible for ensuring that the property is set if required by the underlying storage. This turned out to be handled incompletely in the WebDAV backend. This change moves this into the engine: - UID is now field. It does not get used for matching because the engine cannot rely on it being stored by both sides. - It gets parsed if present, but only generated if explicitly enabled (because that is the traditional behavior). - It is never shown in the DevInf's CtCap because the Synthesis engine would always show it regardless whether a rule enabled the property. That's because rules normally only get triggered after exchanging DevInf and thus DevInf has to be rule-independent. We don't want it shown because then merging the incoming item during a local sync would use the incoming UID, even if it is empty. - Before writing, ensure that UID is set. When updating an existing item, the Synthesis engine reads the existing item, preserves the existing UID unless the peer claims to support UID, and then updates with the existing UID. This works for local sync (where SyncEvolution never claims to support UID when talking to the other side). It will break with peers which have UID in their CtCap although they rewrite the UID and backends whose underlying storage cannot handle UID changes during an update (for example, CardDAV). * engine: flush map items less frequently The Synthesis API does not say explicitly, but in practice all map items get updated in a tight loop. Rewriting the m_mappingNode (case insensitive string comparisons) and serialization to disk (std::ostrstream) consume a significant amount of CPU cycles and cause extra disk writes that can be avoided by making some assumptions about the sequence of API calls and flushing only once. * local sync: exchange SyncML messages via shared memory Encoding/decoding of the uint8_t array in D-Bus took a surprisingly large amount of CPU cycles relative to the rest of the SyncML message processing. Now the actual data resides in memory-mapped temporary files and the D-Bus messages only contain offset and size inside these files. Both sides use memory mapping to read and write directly. For caching 1000 contacts with photos on a fast laptop, total sync time roughly drops from 6s to 3s. To eliminate memory copies, memory handling in libsynthesis or rather, libsmltk is tweaked such that it allocates the buffer used for SyncML message data in the shared memory buffer directly. This relies on knowledge of libsmltk internals, but those shouldn't change and if they do, SyncEvolution will notice ("unexpected send buffer"). * local sync: avoid updating meta data when nothing changed The sync meta data (sync anchors, client change log) get updated after a sync even if nothing changed and the existing meta data could be used again. This can be skipped for local sync, because then SyncEvolution can ensure that both sides skip updating the meta data. With a remote SyncML server that is not possible and thus SyncEvolution has to update its data. It is based on the observation that when the server side calls SaveAdminData, the client has sent its last message and the sync is complete. At that point, SyncEvolution can check whether anything has changed and if not, skip saving the server's admin data and stop the sync without sending the real reply to the client. Instead the client gets an empty message with "quitsync" as content type. Then it takes shortcuts to close down without finalizing the sync engine, because that would trigger writing of meta data changes. The server continues its shutdown normally. This optimization is limited to syncs with a single source, because the assumption about when aborting is possible is harder to verify when multiple sources are involved. * PBAP: support SYNCEVOLUTION_PBAP_CHUNK_TRANSFER_TIME <= 0 When set to 0 or less, the chunk size is not getting adapted at all while still using transfers in chunks. * PBAP: use raw text items This avoids the redundant parse/generate step on the sending side of the PBAP sync. * PBAP syncing: updated photo not always stored Because photo data was treated like a C string, changes after any embedded null byte were ignored during a comparison. * ephemeral sync: don't write binfile client files (FDO #55921) When doing PBAP caching, we don't want any meta data written because the next sync would not use it anyway. With the latest libsynthesis we can configure "/dev/null" as datadir for the client's binfiles and libsynthesis will avoid writing them. The PIM manager uses this for PBAP syncing automatically. For testing it can be enabled by setting the SYNCEVOLUTION_EPHEMERAL env variable. * PBAP: avoid empty field filter Empty field filter is supposed to mean "return all supported fields". This used to work and stopped working with Android phones after an update to 4.3 (seen on Galaxy S3); now the phone only returns the mandatory TEL, FN, N fields. The workaround is to replace the empty filter list with the list of known and supported properties. This means we only pull data we really need, but it also means we won't get to see any additional properties that the phone might support. * PBAP: transfer in chunks (FDO #77272) If enabled via env variables, PullAll transfers will be limited to a certain numbers contacts at different offsets until all data got pulled. See PBAP README for details. When transfering in chunks, the enumeration of contacts for the engine no longer matches the PBAP enumeration. Debug output uses "offset #x" for PBAP and "ID y" for the engine. * PBAP: remove transfer via pipe Using a pipe was never fully supported by obexd (blocks obexd). Transfering in suitably sized chunks (FDO #77272) will be a more obexd friendly solution with a similar effect (not having to buffer the entire address book in memory). * PBAP: Suspend/ResumeSync() (FDO #72112) By default, the new API freezes a sync by stopping to consume data on the local side of the sync. In addition, the information that the sync is freezing is now also handed down to the transport and all sources. In the case of PBAP caching, the local transport notifies the child where the PBAP source then uses Bluez 5.15 Transfer1.Suspend/Resume to freeze/thaw the actual OBEX transfer. If that fails (for example, not implemented because Bluez is too old or the transfer is still queueing), then the transfer gets cancelled and the entire sync fails. This is desirable for PBAP caching and Bluetooth because a failed sync can easily be recovered from (just start it again) and the overall goal is to free up Bluetooth bandwidth quickly. * PBAP: transfer data via pipe (part of FDO #72112) The main advantage is that processed data can be discarded immediately. When using a plain file, the entire address book must be stored in it. The drawback is that obexd does not react well to a full pipe. It simply gets stuck in a blocking write(); in other words, all obexd operations get frozen and obexd stops responding on D-Bus. * PIM: include CardDAV in CreatePeer() This adds "protocol: CardDAV" as a valid value, with corresponding changes to the interpretation of some existing properties and some new ones. The API itself is not changed. Suspending a CardDAV sync is possible. This freezes the internal SyncML message exchange, so data exchange with the CardDAV server may continue for a while after SuspendPeer(). Photo data is always downloaded immediately. The "pbap-sync" flag in SyncPeerWithFlags() has no effect. Syncing can be configured to be one-way (local side is read-only cache) or two-way (local side is read/write). Meta data must be written either way, to speed up caching or allow two-way syncing. The most common case (no changes on either side) will have to be optimized such that existing meta data is not touched and thus no disk writes occur. * PIM: handle SuspendPeer() before and after transfer (FDO #82863) A SuspendPeer() only succeeded while the underlying Bluetooth transfer was active. Outside of that, Bluez errors caused SyncEvolution to attempt a cancelation of the transfer and stopped the sync. When the transfer was still queueing, obexd returns org.bluez.obex.Error.NotInProgress. This is difficult to handle for SyncEvolution: it cannot prevent the transfer from starting and has to let it become active before it can suspend the transfer. Canceling would lead to difficult to handle error cases (like partially parsed data) and therefore is not done. The Bluez team was asked to implement suspending of queued transfers (see "org.bluez.obex.Transfer1 Suspend/Resume in queued state" on linux-bluetooth@vger.kernel.org), so this case might not happen anymore with future Bluez. When the transfer completes before obexd processes the Suspend(), org.freedesktop.DBus.Error.UnknownObject gets returned by obexd. SyncEvolution can ignore errors which occur after the active transfer completed. In addition, it should prevent starting the next one. This may be relevant for transfer in chunks, although the sync engine will also stop asking for data and thus typically no new transfer gets triggered anyway. * PIM: add suspend/resume/abort to sync.py CTRL-C while waiting for the end of a sync causes an interactive prompt to appear where one can choose been suspend/resume/abort and continuing to wait. CTRL-C again in the prompt aborts the script. * PIM: enhanced progress notifications (FDO #72114) This adds GetPeerStatus() and "progress" events. Progress is reported based on the "item received" Synthesis event and the total item count. A modified libsynthesis is needed where the SyncML binfile client on the target side of the local sync actually sends the total item count (via NumberOfChanges). This cannot be done yet right at the start of the sync, only the second SyncML message will have it. That is acceptable, because completion is reached very quickly anyway for syncs involving only one message. At the moment, SyncContext::displaySourceProgress() holds back "item received" events until a different event needs to be emitted. Progress reporting might get more fine-grained when adding allowing held back events to be emitted at a fixed rate, every 0.1s. This is not done yet because it seems to work well enough already. For testing and demonstration purposes, sync.py gets command line arguments for setting progress frequency and showing progress either via listening to signals or polling. * PIM: add SyncPeerWithFlags() and 'pbap-sync' flag (FDO #70950) The is new API and flag grant control over the PBAP sync mode. * PIM: fix phone number normalization The parsed number always has a country code, whereas SyncEvolution expected it to be zero for strings without an explicit country code. This caused a caller ID lookup of numbers like "089788899" in DE to find only telephone numbers in the current default country, instead of being more permissive and also finding "+189788899". The corresponding unit test was broken and checked for the wrong result. Found while investigating an unrelated test failure when updating libphonenumber. * engine: enable batching by default (FDO #52669) This reverts commit c435e937cd406e904c437eec51a32a6ec6163102. Commit 7b636720a in libsynthesis fixes an unitialized memory read in the asynchronous item update code path. Testing confirms that we can now used batched writes reliably with EDS (the only backend currently supporting asynchronous writes + batching), so this change enables it again also for local and SyncEvolution<->SyncEvolution sync (with asynchronous execution of contact add/update overlapped with SyncML message exchanges) and other SyncML syncs (with changes combined into batches and executed at the end of each message). * Various compiler problems and warnings fixed; compiles with --with-warnings=fatal on current Debian Testing and Ubuntu Trusty (FDO #79316). * D-Bus server: fix unreliable shutdown handling Occassionally, syncevo-dbus-server locked up after receiving a CTRL-C. This primarily affected nightly testing, in particular (?) on Ubuntu Lucid. * D-Bus: use streams for direct IPC with GIO When using GIO, it is possible to avoid the DBusServer listening on a publicly accessible address. Connection setup becomes more reliable, too, because the D-Bus server side can detect that a child died because the connection will be closed. When using libdbus, the traditional server/listen and client/connect model is still used. * LogRedirect: safeguard against memory corruption When aborting, our AbortHandler gets called to close down logging. This may involve memory allocation, which is unsafe. In FDO #76375, a deadlock on a libc mutex was seen. To ensure that the process shuts down anyway, install an alarm and give the process five seconds to shut down before the SIGALRM signal will kill it. Upgrading from releases <= 1.3.99.4: If the value of "username/databaseUser/proxyUser" contains a colon, the "user:" prefix must be added to the value, to continue treating it like a plain user name and not some reference to an unknown identity provider (like "id:", "goa:", "signon:", etc.). The lookup of passwords in GNOME Keyring was updated slightly in 1.3.99.5. It may be necessary to set passwords anew if the old one is no longer found. Upgrading from release 1.2.x: The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution --configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the "refresh-from-server" sync mode to avoid getting throttled with 417 'retry later' errors. The same must be added to existing configs manually: syncevolution --configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2: Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. SyncEvolution 1.4.99.4 -> 1.5, 31.10.2014 ========================================= Mostly a bug fix release. Details: * vcard: fix caching of PBAP contacts (FDO #84710) After changing PBAP to send raw items, caching them led to unnecessary disk writes and bogus "contacts changed" reports. That's because the merge script relied on the exact order of properties, which was only the same when doing the redundant decode/encode on the PBAP side. Instead of reverting back to sending re-encoded items, better enhance the contact merge script such that it detects contacts as unchanged when just the order of entries in the property arrays is different. This relies on an enhanced libsynthesis with the new RELAXEDCOMPARE() and modified MERGEFIELDS(). * sync: ignore unnecessary username property A local sync or Bluetooth sync do not need the 'username' property. When it is set despite that, issue a warning. Previously, the value was checked even when not needed, which caused such syncs to fail when set to something other than a plain username. * D-Bus server: fix unreliable shutdown handling Occassionally, syncevo-dbus-server locked up after receiving a CTRL-C. This primarily affected nightly testing, in particular (?) on Ubuntu Lucid. * scripting: prevent premature loop timeouts The more complex "avoid data loss during merging" scripting ran for longer than 5s limit under extreme conditions (full logging, busy system, running under valgrind), which resulted in aborting the script and a 10500 "local internal error" sync failure. * signon: fix providersignon.so (TC-1667) The shared providersignon.so ended up being compiled with "gsso" as prefix for the username. There also was a problem with invalid reference counting. * PBAP: support SYNCEVOLUTION_PBAP_CHUNK_TRANSFER_TIME <= 0 When set to 0 or less, the chunk size is not getting adapted at all while still using transfers in chunks. SyncEvolution 1.4.99.3 -> 1.4.99.4, 10.09.2014 ============================================== One focus in this release was on minimizing CPU consumption and disk writes. The most common case, a two-way sync with no changes on either side, no longer rewrites any meta data files. CPU consumption during local sync was reduced to one third by exchanging messages via shared memory instead of internal D-Bus. Redundant vCard decode/encode on the sending side of PBAP and too agressive flushing of meta data during a normal sync were removed. Altogether, sending 1000 contacts with photo data in a refresh-from-server local sync takes only one sixth of the CPU cycles compared to 1.3.99.3 (measured with valgrind's callgrind on x86_64). Based on community feedback and discussions, the terminology used in SyncEvolution for configuration, local sync and database access was revised. Some usability issues with setting up access to databases were addressed. For Google, the obsolete SyncML config template was removed and CalDAV/CardDAV were merged into a single "Google" template. Using Google Calendar/Contacts with OAuth2 authentication on a headless server becomes a bit easier: it is possible to set up access on one system with a GUI using either gSSO or GNOME Online Accounts, then take the OAuth2 refresh token and use it in SyncEvolution on a different system. See http://cgit.freedesktop.org/SyncEvolution/syncevolution/tree/src/backends/oauth2/README Some issues accessing Apple iCloud were fixed such that CardDAV works by just giving SyncEvolution username=foobar@icloud.com and password. No throrough testing was done, so iCloud support is still experimental. The PIM Manager API also supports Google Contact syncing. Some problems with suspending a PBAP sync were fixed. Suspend/abort can be tested with the sync.py example. The EDS memo backend is able to switch between syncing in plain text and iCalendar 2.0 VJOURNAL automatically. Details: * oauth2: new backend using libsoup/libcurl New backend implements identity provider for obtaining OAuth2 access token for systems without HMI support. Access token is obtained by making direct HTTP request to OAuth2 server and using refresh token obtained by user in some other way. New provider automatically updates stored refresh token when OAuth2 server is issuing new one. * PBAP: use raw text items This avoids the redundant parse/generate step on the sending side of the PBAP sync. * datatypes: raw text items with minimal conversion (FDO #52791) When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution "databaseFormat", all parsing and conversion is skipped. The backend's data is identical to the item data in the engine. Finding duplicates in a slow sync is very limited when using these types because the entire item data must match exactly. This is useful for the file backend when the goal is to store an exact copy of what a peer has or for limited, read-only backends (PBAP). The downside of using the raw types is that the peer is not given accurate information about which vCard or iCalendar properties are supported, which may cause some peers to not send all data. * engine: flush map items less frequently The Synthesis API does not say explicitly, but in practice all map items get updated in a tight loop. Rewriting the m_mappingNode (case insensitive string comparisons) and serialization to disk (std::ostrstream) consume a significant amount of CPU cycles and cause extra disk writes that can be avoided by making some assumptions about the sequence of API calls and flushing only once. * SoupTransport: drop CA file check It used to be necessary to specify a CA file for libsoup to enable SSL certificate checking. Nowadays libsoup uses the default CA store unless told otherwise, so the check in SyncEvolution became obsolete. However, now there is a certain risk that no SSL checking is done although the user asked for it (when libsoup is not recent enough or compiled correctly). * local sync: exchange SyncML messages via shared memory Encoding/decoding of the uint8_t array in D-Bus took a surprisingly large amount of CPU cycles relative to the rest of the SyncML message processing. Now the actual data resides in memory-mapped temporary files and the D-Bus messages only contain offset and size inside these files. Both sides use memory mapping to read and write directly. For caching 1000 contacts with photos on a fast laptop, total sync time roughly drops from 6s to 3s. To eliminate memory copies, memory handling in libsynthesis or rather, libsmltk is tweaked such that it allocates the buffer used for SyncML message data in the shared memory buffer directly. This relies on knowledge of libsmltk internals, but those shouldn't change and if they do, SyncEvolution will notice ("unexpected send buffer"). * local sync: avoid updating meta data when nothing changed The sync meta data (sync anchors, client change log) get updated after a sync even if nothing changed and the existing meta data could be used again. This can be skipped for local sync, because then SyncEvolution can ensure that both sides skip updating the meta data. With a remote SyncML server that is not possible and thus SyncEvolution has to update its data. This optimization is only used for local syncs with one source. It is based on the observation that when the server side calls SaveAdminData, the client has sent its last message and the sync is complete. At that point, SyncEvolution can check whether anything has changed and if not, skip saving the server's admin data and stop the sync without sending the real reply to the client. Instead the client gets an empty message with "quitsync" as content type. Then it takes shortcuts to close down without finalizing the sync engine, because that would trigger writing of meta data changes. The server continues its shutdown normally. This optimization is limited to syncs with a single source, because the assumption about when aborting is possible is harder to verify when multiple sources are involved. * PIM: include CardDAV in CreatePeer() This adds "protocol: CardDAV" as a valid value, with corresponding changes to the interpretation of some existing properties and some new ones. The API itself is not changed. Suspending a CardDAV sync is possible. This freezes the internal SyncML message exchange, so data exchange with the CardDAV server may continue for a while after SuspendPeer(). Photo data is always downloaded immediately. The "pbap-sync" flag in SyncPeerWithFlags() has no effect. Syncing can be configured to be one-way (local side is read-only cache) or two-way (local side is read/write). Meta data must be written either way, to speed up caching or allow two-way syncing. The most common case (no changes on either side) will have to be optimized such that existing meta data is not touched and thus no disk writes occur. * PIM: handle SuspendPeer() before and after transfer (FDO #82863) A SuspendPeer() only succeeded while the underlying Bluetooth transfer was active. Outside of that, Bluez errors caused SyncEvolution to attempt a cancelation of the transfer and stopped the sync. When the transfer was still queueing, obexd returns org.bluez.obex.Error.NotInProgress. This is difficult to handle for SyncEvolution: it cannot prevent the transfer from starting and has to let it become active before it can suspend the transfer. Canceling would lead to difficult to handle error cases (like partially parsed data) and therefore is not done. The Bluez team was asked to implement suspending of queued transfers (see "org.bluez.obex.Transfer1 Suspend/Resume in queued state" on linux-bluetooth@vger.kernel.org), so this case might not happen anymore with future Bluez. When the transfer completes before obexd processes the Suspend(), org.freedesktop.DBus.Error.UnknownObject gets returned by obexd. SyncEvolution can ignore errors which occur after the active transfer completed. In addition, it should prevent starting the next one. This may be relevant for transfer in chunks, although the sync engine will also stop asking for data and thus typically no new transfer gets triggered anyway. * PIM: add suspend/resume/abort to sync.py CTRL-C while waiting for the end of a sync causes an interactive prompt to appear where one can choose been suspend/resume/abort and continuing to wait. CTRL-C again in the prompt aborts the script. * PIM: fix sync.py --sync-flags The help text used single quotes for the JSON example instead of the required double quotes. Running without --sync-flags was broken because of trying to parse the empty string as JSON. * command line: revise usability checking of datastores When configuring a new sync config, the command line checks whether a datastore is usable before enabling it. If no datastores were listed explicitly, only the usable ones get enabled. If unusable datastores were explicitly listed, the entire configure operation fails. This check was based on listing databases, which turned out to be too unspecific for the WebDAV backend: when "database" was set to some URL which is good enough to list databases, but not a database URL itself, the sources where configured with that bad URL. Now a new SyncSource::isUsable() operation is used, which by default just falls back to calling the existing Operations::m_isEmpty. In practice, all sources either check their config in open() or the m_isEmpty operation, so the source is usable if no error is enountered. For WebDAV, the usability check is skipped because it would require contacting a remote server, which is both confusing (why does a local configure operation need the server?) and could fail even for valid configs (server temporarily down). The check was incomplete anyway because listing databases gave a fixed help text response when no credentials were given. For usability checking that should have resulted in "not usable" and didn't. The output during the check was confusing: it always said "listing databases" without giving a reason why that was done. The intention was to give some feedback while a potentially expensive operation ran. Now the isUsable() method itself prints "checking usability" if (and only if!) such a check is really done. Sometimes datastores were checked even when they were about to be configure as "disabled" already. Now checking such datastores is skipped. * EDS: memo syncing as iCalendar 2.0 (FDO #52714) When syncing memos with a peer which also supports iCalendar 2.0 as data format, the engine will now pick iCalendar 2.0 instead of converting to/from plain text. The advantage is that some additional properties like start date and categories can also be synchronized. The code is a lot simpler, too, because the EDS specific iCalendar 2.0 <-> text conversion code can be removed. * datatypes: text/calendar+plain revised heuristic When sending a VEVENT, DESCRIPTION was set to the SUMMARY if empty. This may have been necessary for some peers, but for notes (= VJOURNAL) we don't know that (hasn't been used in the past) and don't want to alter the item unnecessarily, so skip that part and allow empty DESCRIPTION. When receiving a plain text note, the "text/calendar+plain" type used to store the first line as summary and the rest as description. This may be correct in some cases and wrong in others. The EDS backend implemented a different heuristic: there the first line is copied into the summary and stays in the description. This makes a bit more sense (the description alone is always enough to understand the note). Therefore and to avoid behavioral changes for EDS users when switching the EDS backend to use text/calendar+plain, the engine now uses the same approach. * source -> datastore rename, improved terminology The word "source" implies reading, while in fact access is read/write. "datastore" avoids that misconception. Writing it in one word emphasizes that it is single entity. While renaming, also remove references to explicit --*-property parameters. The only necessary use today is "--sync-property ?" and "--datastore-property ?". --datastore-property was used instead of the short --store-property because "store" might be mistaken for the verb. It doesn't matter that it is longer because it doesn't get typed often. --source-property must remain valid for backward compatility. As many user-visible instances of "source" as possible got replaced in text strings by the newer term "datastore". Debug messages were left unchanged unless some regex happened to match it. The source code will continue to use the old variable and class names based on "source". Various documentation enhancements: Better explain what local sync is and how it involves two sync configs. "originating config" gets introduces instead of just "sync config". Better explain the relationship between contexts, sync configs, and source configs ("a sync config can use the datastore configs in the same context"). An entire section on config properties in the terminology section. "item" added (Todd Wilson correctly pointed out that it was missing). Less focus on conflict resolution, as suggested by Graham Cobb. Fix examples that became invalid when fixing the password storage/lookup mechanism for GNOME keyring in 1.4. The "command line conventions", "Synchronization beyond SyncML" and "CalDAV and CardDAV" sections were updated. It's possible that the other sections also contain slightly incorrect usage of the terminology or are simply out-dated. * local sync: allow config name in syncURL=local:// Previously, only syncURL=local://@ was allowed and used the "target-config@context name" config as target side in the local sync. Now "local://config-name@context-name" or simply "local://config-name" are also allowed. "target-config" is still the fallback if only a context is give. It also has one more special meaning: "--configure target-config@google" will pick the "Google" template automatically because it knows that the intention is to configure the target side of a local sync. It does not know that when using some other name for the config, in which case the template (if needed) must be specified explicitly. The process name in output from the target side now also includes the configuration name if it is not the default "target-config". * Google: remove SyncML template, combine CalDAV/CardDAV Google has turned off their SyncML server, so the corresponding "Google Contacts" template became useless and needs to be removed. It gets replaced by a "Google" template which combines the three different URLs currently used by Google for CalDAV/CardDAV. This new template can be used to configure a "target-config@google" with default calendar and address book database already enabled. The actual URL of these databases will be determined during the first sync using them. The template relies on the WebDAV backend's new capability to search multiple different entries in the syncURL property for databases. To avoid listing each calendar twice (once for the legacy URL, once with the new one) when using basic username/password authentication, the backend needs a special case for Google and detect that the legacy URL does not need to be checked. * config: allow storing credentials for email address When configuring a WebDAV server with username = email address and no URL (which is possible if the server supports service discovery via the domain in the email address), then storing the credentials in the GNOME keyring used to fail with "cannot store password in GNOME keyring, not enough attributes". That is because GNOME keyring seemed to get confused when a network login has no server name and some extra safeguards were added to SyncEvolution to avoid this. To store the credentials in the case above, the email address now gets split into user and domain part and together get used to look up the password. SyncEvolution 1.4.99.2 -> 1.4.99.3, 23.07.2014 ============================================== This release enhances CalDAV/CardDAV and PBAP syncing and fixes some problems. The enhanced conflict handling introduced 1.4.99.2 was unintentionally limited to syncs with EDS on the server side; it is now also available for example in WebDAV<->SyncML bridge setups. Details: * CardDAV: implement read-ahead Instead of downloading contacts one-by-one with GET, SyncEvolution now looks at contacts that are most likely going to be needed soon and gets all of them at once with addressbook-multiget REPORT. The number of contacts per REPORT is 50 by default, configurable by setting the SYNCEVOLUTION_CARDDAV_BATCH_SIZE env variable. This has two advantages: - It avoids round-trips to the server and thus speeds up a large download (100 small contacts with individual GETs took 28s on a fast connection, 3s with two REPORTs). - It reduces the overall number of requests. Google CardDAV is known to start issuing intermittent 401 authentication errors when the number of contacts retrieved via GET is too large. Perhaps this can be avoided with addressbook-multiget. * Google Calendar: remove child hack, improve alarm hack (FDO #63881) Google recently enhanced support for RECURRENCE-ID, so SyncEvolution no longer needs to replace the property when uploading a single detached event with RECURRENCE-ID. However, several things are still broken in the server, with no workaround in SyncEvolution: - Removing individual events gets ignored by the server; a full "wipe out server data" might work (untested). - When updating the parent event, all child events also get updated even though they were included unchanged in the data sent by SyncEvolution. - The RECURRENCE-ID of a child event of an all-day recurring event does not get stored properly. - The update hack seems to fail for complex meetings: uploading them once and then deleting them seems to make uploading them again impossible. All of these issues were reported to Google and are worked on there, so perhaps the situation will improve. In the meantime, syncing with Google CalDAV should better be limited to: - Downloading a Google calendar in one-way mode. - Two-way syncing of simple calendars without complex meeting serieses. While updating the Google workarounds, the alarm hack (sending a new event without alarms twice to avoid the automatic server side alarm) was simplified. Now the new event gets sent only once with a pseudo-alarm. * ephemeral sync: don't write binfile client files (FDO #55921) When doing PBAP caching, we don't want any meta data written because the next sync would not use it anyway. With the latest libsynthesis we can configure "/dev/null" as datadir for the client's binfiles and libsynthesis will avoid writing them. The PIM manager uses this for PBAP syncing automatically. For testing it can be enabled by setting the SYNCEVOLUTION_EPHEMERAL env variable. * PBAP: avoid empty field filter Empty field filter is supposed to mean "return all supported fields". This used to work and stopped working with Android phones after an update to 4.3 (seen on Galaxy S3); now the phone only returns the mandatory TEL, FN, N fields. The workaround is to replace the empty filter list with the list of known and supported properties. This means we only pull data we really need, but it also means we won't get to see any additional properties that the phone might support. * PBAP: transfer in chunks (FDO #77272) If enabled via env variables, PullAll transfers will be limited to a certain numbers contacts at different offsets until all data got pulled. See PBAP README for details. When transfering in chunks, the enumeration of contacts for the engine no longer matches the PBAP enumeration. Debug output uses "offset #x" for PBAP and "ID y" for the engine. * PBAP: remove transfer via pipe Using a pipe was never fully supported by obexd (blocks obexd). Transfering in suitably sized chunks (FDO #77272) will be a more obexd friendly solution with a similar effect (not having to buffer the entire address book in memory). * engine: enable batching by default (FDO #52669) This reverts commit c435e937cd406e904c437eec51a32a6ec6163102. Commit 7b636720a in libsynthesis fixes an unitialized memory read in the asynchronous item update code path. Testing confirms that we can now used batched writes reliably with EDS (the only backend currently supporting asynchronous writes + batching), so this change enables it again also for local and SyncEvolution<->SyncEvolution sync (with asynchronous execution of contact add/update overlapped with SyncML message exchanges) and other SyncML syncs (with changes combined into batches and executed at the end of each message). * datatypes: fix contact caching Adding grouping to the contact datatype in 1.4.99.2 broke PBAP caching: when sending an empty URL, for example, during the sync, the parsed contact had different field arrays than the locally stored contact, because the latter was saved without the empty URL. This caused the field-based comparison to detect a difference even when the final, reencoded contact wasn't different at all. To solve this, syncing now uses the same "don't send empty properties" configuration as local storages. Testing shows that this resolves the difference for EDS. A more resilient solution would be to add a check based on the encoded data, but that's more costly performance wise. * datatypes: fix vCard handling The new "preserve repeating properties during conflict resolution" feature from 1.4.99.2 was only active when using EDS as storage. The relevant merge script must be applied to all datatypes, not just the EDS flavor. The feature was also unintentionally active when running in caching mode. This caused two problems: - The cached item was updated even though only the ordering of repeating properties had been modified during merging. - The merged item was sent back to the client side, which was undesirable (caching is supposed to be one-way) or even impossible (PBAP is read-only, causing sync failures eith error 20030). We must check for caching mode and disable merging when it is active. We also must not tell the engine that we updated the photo property in the winning item, because then that item would get sent to the read-only side of the sync. Perhaps a better solution would be to actually tell the engine that the remote side is read-only when we activate caching mode. * datatypes: avoid PHOTO corruption during merge (FDO #77065) When handling an update/update conflict (both sides of the sync have an updated contact) and photo data was moved into a local file by EDS, the engine merged the file path and the photo data together and thus corrupted the photo. The engine does not know about the special role of the photo property. This needs to be handled by the merge script, and that script did not cover this particular situation. Now the loosing side is cleared, causing the engine to then copy the winning side over into the loosing one. Found by Renato Filho/Canonical when testing SyncEvolution for Ubuntu 14.04. * PBAP syncing: updated photo not always stored Because photo data was treated like a C string, changes after any embedded null byte were ignored during a comparison. * PIM: fix phone number normalization The parsed number always has a country code, whereas SyncEvolution expected it to be zero for strings without an explicit country code. This caused a caller ID lookup of numbers like "089788899" in DE to find only telephone numbers in the current default country, instead of being more permissive and also finding "+189788899". The corresponding unit test was broken and checked for the wrong result. Found while investigating an unrelated test failure when updating libphonenumber. * Various compiler problems and warnings fixed; compiles with --with-warnings=fatal on current Debian Testing and Ubuntu Trusty (FDO #79316). SyncEvolution 1.4.99.1 -> 1.4.99.2, 23.05.2014 ============================================== 1.4.99.2 enhances interoperability with CardDAV servers and in particular Google Contacts considerably. Contact data gets converted to and from the format typically used by CardDAV servers, so now anniversary, spouse, manager, assistant and instant message information are exchanged properly. Custom labels get stored in EDS as extensions and no longer get lost when updating some other aspects of a contact. However, Evolution does not show custom labels and removes them when editing a property which has a custom label (BGO #730636). Scanning for CardDAV/CalDAV resources was enhanced. It now finds additional calendars with Google CalDAV and works with iCloud. However, syncing with iCloud ran into a server bug (reported as 17001498 "CalDAV REPORT drops calendar data") and needs further work. Details: * vcard profile: avoid data loss during merging When resolving a merge conflict, repeating properties were taken wholesale from the winning side (for example, all email addresses). If a new email address had been added on the loosing side, it got lost. Arguably it is better to preserve as much data as possible during a conflict. SyncEvolution now does that in a merge script by checking which properties in the loosing side do not exist in the winning side and copying those entries. Typically only the main value (email address, phone number) is checked and not the additional meta data (like the type). Otherwise minor differences (for example, both sides have same email address, but with different types) would lead to duplicates. Only addresses are treated differently: for them all attributes (street, country, city, etc.) are compared, because there is no single main value. * engine: UID support in contact data Before, the UID property in a vCard was ignored by the engine. Backends were responsible for ensuring that the property is set if required by the underlying storage. This turned out to be handled incompletely in the WebDAV backend. This change moves this into the engine: - UID is now field. It does not get used for matching because the engine cannot rely on it being stored by both sides. - It gets parsed if present, but only generated if explicitly enabled (because that is the traditional behavior). - It is never shown in the DevInf's CtCap because the Synthesis engine would always show it regardless whether a rule enabled the property. That's because rules normally only get triggered after exchanging DevInf and thus DevInf has to be rule-independent. We don't want it shown because then merging the incoming item during a local sync would use the incoming UID, even if it is empty. - Before writing, ensure that UID is set. When updating an existing item, the Synthesis engine reads the existing item, preserves the existing UID unless the peer claims to support UID, and then updates with the existing UID. This works for local sync (where SyncEvolution never claims to support UID when talking to the other side). It will break with peers which have UID in their CtCap although they rewrite the UID and backends whose underlying storage cannot handle UID changes during an update (for example, CardDAV). * CardDAV: use Apple/Google/CardDAV vCard flavor In principle, CardDAV servers support arbitrary vCard 3.0 data. Extensions can be different and need to be preserved. However, when multiple different clients or the server's Web UI interpret the vCards, they need to agree on the semantic of these vCard extensions. In practice, CardDAV was pushed by Apple and Apple clients are probably the most common clients of CardDAV services. When the Google Contacts Web UI creates or edits a contact, Google CardDAV will send that data using the vCard flavor used by Apple. Therefore it makes sense to exchange contacts with *all* CardDAV servers using that format. This format could be made configurable in SyncEvolution on a case-by-case basis; at the moment, it is hard-coded. During syncing, SyncEvolution takes care to translate between the vCard flavor used internally (based on Evolution) and the CardDAV vCard flavor. This mapping includes: X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE Any IMPP property declared as X-SERVICE-TYPE=AIM will get mapped to X-AIM. Same for others. Some IMPP service types have no known X- property extension; they are stored in EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE (for example, GaduGadu and Groupwise) are stored with X-SERVICE-TYPE values chosen by SyncEvolution so that Google CardDAV preserves them (GroupWise with mixed case got translated by Google into Groupwise, so the latter is used). Google always sends an X-ABLabel:Other for IMPP. This is ignored because the service type overrides it. The value itself also gets transformed during the mapping. IMPP uses an URI as value, with a chat protocol (like "aim" or "xmpp") and some protocol specific identifier. For each X- extension the protocol is determined by the property name and the value is the protocol specific identifier without URL encoding. X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel The mapping is based on the X-ABLabel property attached to the X-ABRELATEDNAMES property. This depends on the English words "Spouse", "Manager", "Assistant" that Google CardDAV and Apple devices seem to use regardless of the configured language. As with IMPP, only the subset of related names which have a corresponding X- property extension get mapped. The rest is stored in EDS using the X-ABRELATEDNAMES property. X-ANNIVERSARY <-> X-ABDATE Same here, with X-ABLabel:Anniversary as the special case which gets mapped. X-ABLabel parameter <-> property CardDAV vCards have labels attached to arbitrary other properties (TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags: item1.X-ABDATE:2010-01-01 item1.X-ABLabel:Anniversary The advantage is that property values can contain arbitrary characters, including line breaks and double quotation marks, which is not possible in property parameters. Neither EDS nor KDE (judging from the lack of responses on the KDE-PIM mailing list) support custom labels. SyncEvolution could have used grouping as it is done in CardDAV, but grouping is not used much (not at all?) by the UIs working with the vCards in EDS and KDE. It seemed easier to use a new X-ABLabel parameter. Characters which cannot be stored in a parameter get converted (double space to single space, line break to space, etc.) during syncing. In practice, these characters don't appear in X-ABLabel properties anyway because neither Apple nor Google UIs allow entering them for custom labels. The "Other" label is used by Google even in case where it adds no information. For example, all XMPP properties have an associated X-ABLabel=Other although the Web UI does not provide a means to edit or show such a label. Editing the text before the value in the UI changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for other fields. Therefore the "Other" label is ignored by removing it during syncing. X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the order of properties in the UI) gets stored in CardDAV. The only exception is Google CardDAV which got confused when an IMPP property had both X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google, X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering of chat information can get lost when syncing with Google. * synccompare: support grouping and quoted parameter strings Grouped properties are sorted first according to the actual property name, then related properties are moved to the place where their group tag appears first. The first grouped property gets a "- " prefix, all following ones are just indended with " ". The actual group tag is not part of the normalized output, because its value is irrelevant: BDAY:19701230 - EMAIL:john@custom.com X-ABLabel:custom-label2 ... FN:Mr. John 1 Doe Sr. - IMPP;X-SERVICE-TYPE=AIM:aim:aim X-ABLabel:Other ... - X-ABDATE:19710101 X-ABLabel:Anniversary Redundant tags (those set for only a single property, X-ABLabel:Other) get removed as part of normalizing an item. * WebDAV: use server's order when listing collections When doing a recursive scan of the home set, preserve the order of entries as reported by the server and check the first one first. The server knows better which entries are more relevant for the user (and thus should be the default) or may have some other relevant order. Previously, SyncEvolution replaced that order with sorting by URL, which led to a predictable, but rather meaningless order. For example, Google lists the users own calendar first, followed by the shared calendars sorted alphabetical by their name. Now SyncEvolution picks the main calendar as default correctly when scanning from https://www.google.com/calendar/dav/. * WebDAV: improved database search (Google, Zimbra) Zimbra has a principal URL that also serves as home set. When using it as start URL, SyncEvolution only looked the URL once, without listing its content, and thus did not find the databases. When following the Zimbra principal URL indirectly, SyncEvolution did check all of the collections there recursively. Unfortunately that also includes many mail folders, causing the scan to abort after checking 1000 collections (an internal safe guard). The solution for both includes tracking what to do with a URL. For the initial URL, only meta data about the URL itself gets checked. Recursive scanning is only done for the home set. If that home set contains many collections, scanning is still slow and may run into the internal safe guard limit. This cannot be avoided because the CalDAV spec explicitly states that the home set may contain normal collections which contain other collections, so a client has to do the recursive scan. When looking at a specific calendar, Google CalDAV does not report what the current principal or the home set is and therefore SyncEvolution stopped after finding just the initial calendar. Now it detects the lack of meta information and adds all parents also as candidates that need to be looked at. The downside of this is that it doesn't know anything about which parents are relevant, so it ends up checking https://www.google.com/calendar/ and https://www.google.com/. In both cases Basic Auth gets rejected with a temporary redirect to the Google login page, which is something that SyncEvolution must ignore immediately during scanning without applying the resend workaround for "temporary rejection of valid credentials" that can happen for valid Google CalDAV URLs. * WebDAV: enhanced database search (Google Calendar) Additional databases where not found for several reasons. SyncEvolution ignored all shared calendars (http://calendarserver.org/ns/shared) and Google marks the additional calendars that way. The other problem was that the check for leaf collections (= collections which cannot contain other desired collections) incorrectly excluded those collections instead of only preventing listing of their content. With this change, https://www.google.com/calendar/dav/?SyncEvolution=Google can be used as starting point for Google Calendar. * WebDAV: fix database scan on iCloud The calendar home set URL on iCloud (the one ending in /calendars/) is declared as containing calendar data. That was enough for SyncEvolution to accept it incorrectly as calendar. However, the home set only contains calendar data indirectly. * WebDAV: support redirects between hosts and DNS SRV lookup based on URL When finding a new URL, we must be prepared to reinitialize the Neon session with the new host settings. iCloud does not have .well-known support on its www.icloud.com server. To support lookup with a non-icloudd.com email address, we must do DNS SRV lookup when access to .well-known URLs fails. We do this without a www prefix on the host first, because that is what happens to work for icloud.com. With these changes it becomes possible to do database scans on Apple iCloud, using syncURL=https://www.icloud.com or syncURL=https://icloud.com. Giving the syncURL like this is only necessary for a username that does not end in @icloud.com. When the syncURL is not set, the domain for DNS SRV lookup is taken from the username. * WebDAV: more efficient item creation PUT has the disadvantage that a client needs to choose a name and then figure out what the real name on the server is. With Google CardDAV that requires sending another request and only works because the server happens to remember the original name (which is not guaranteed!). POST works for new items without a name and happens to be implemented by Google such that the response already includes all required information (new name and revision string). POST is checked for as described in RFC 5995 once before creating a new item. Servers which don't support it continue to get a PUT. * WebDAV: send "User-Agent: SyncEvolution" Apple iCloud servers reject requests unless they contain a User-Agent header. The exact value doesn't seem to matter. Making the string configurable might be better, but can still be done later when it is more certain whether and for what it is needed. * WebDAV: refactor and fix DNS SRV lookup The syncevo-webdav-lookup script was not packaged. It did not report "not found" DNS results correctly and the caller did not check for this either, so when looking up the information for a domain which does not have DNS SRV entries, SyncEvolution ended up retrying for while as if there had been a temporary lookup problem. * signon: make Accounts optional The new "signon" provider only depends on lib[g]signon-glib. It uses gSSO if found, else UOA. Instead of pulling parameters and the identity via libaccounts-glib, the user of SyncEvolution now has to ensure that the identity exists and pass all relevant parameters in the "signon:" username. * gSSO: adapt to gSSO >= 2.0 * config templates: Funambol URLs Funambol turned of the URL redirect from my.funambol.com to onemedia.com. The Funambol template now uses the current URL. Users with existing Funambol configs must updated the syncURL property manually to https://onemediahub.com/sync Kudos to Daniel Clement for reporting the change. * command line: fix --update from directory The "--update " operation was supposed to take the item luids from the file names inside the directory. That part had not been implemented, turning the operation accidentally into an "--import". Also missing was the escaping/unescaping of luids. Now the same escaping is done as in command line output and command line parsing to make the luids safe for use as file name. * testing: added server-specific tests for CardDAV covering remote item formats and edit conflicts. SyncEvolution 1.4.1 -> 1.4.99.1, 01.04.2014 =========================================== 1.4.99.1 includes several PIM Manager improvements plus some unrelated fixes. Details: * LogRedirect: safeguard against memory corruption When aborting, our AbortHandler gets called to close down logging. This may involve memory allocation, which is unsafe. In FDO #76375, a deadlock on a libc mutex was seen. To ensure that the process shuts down anyway, install an alarm and give the process five seconds to shut down before the SIGALRM signal will kill it. * PBAP: Suspend/ResumeSync() (FDO #72112) By default, the new API freezes a sync by stopping to consume data on the local side of the sync. In addition, the information that the sync is freezing is now also handed down to the transport and all sources. In the case of PBAP caching, the local transport notifies the child where the PBAP source then uses Bluez 5.15 Transfer1.Suspend/Resume to freeze/thaw the actual OBEX transfer. If that fails (for example, not implemented because Bluez is too old or the transfer is still queueing), then the transfer gets cancelled and the entire sync fails. This is desirable for PBAP caching and Bluetooth because a failed sync can easily be recovered from (just start it again) and the overall goal is to free up Bluetooth bandwidth quickly. * PBAP: transfer data via pipe (part of FDO #72112) The main advantage is that processed data can be discarded immediately. When using a plain file, the entire address book must be stored in it. The drawback is that obexd does not react well to a full pipe. It simply gets stuck in a blocking write(); in other words, all obexd operations get frozen and obexd stops responding on D-Bus. * PIM: enhanced progress notifications (FDO #72114) This adds GetPeerStatus() and "progress" events. Progress is reported based on the "item received" Synthesis event and the total item count. A modified libsynthesis is needed where the SyncML binfile client on the target side of the local sync actually sends the total item count (via NumberOfChanges). This cannot be done yet right at the start of the sync, only the second SyncML message will have it. That is acceptable, because completion is reached very quickly anyway for syncs involving only one message. At the moment, SyncContext::displaySourceProgress() holds back "item received" events until a different event needs to be emitted. Progress reporting might get more fine-grained when adding allowing held back events to be emitted at a fixed rate, every 0.1s. This is not done yet because it seems to work well enough already. For testing and demonstration purposes, sync.py gets command line arguments for setting progress frequency and showing progress either via listening to signals or polling. * PIM: add SyncPeerWithFlags() and 'pbap-sync' flag (FDO #70950) The is new API and flag grant control over the PBAP sync mode. * D-Bus: use streams for direct IPC with GIO When using GIO, it is possible to avoid the DBusServer listening on a publicly accessible address. Connection setup becomes more reliable, too, because the D-Bus server side can detect that a child died because the connection will be closed. When using libdbus, the traditional server/listen and client/connect model is still used. * sync output: hide ": started" INFO messages These messages get printed at the start of processing each SyncML message. This is not particularly useful and just adds noise to the output. * signon: fix build Static build was broken for gSSO and UOA (wrong path name to .la file) and gSSO was not enabled properly (wrong condition check). SyncEvolution 1.4 -> 1.4.1, 31.03.2014 ====================================== The first bug fix release in the 1.4 series addresses some issues which occurred on some systems. Details: * EDS: only load one backend plugin of each kind SyncEvolution was meant to load the syncecal or syncebook shared object which uses the most recent libraries (libical, libecal/libebook) on the system and then stop loooking for alternatives. Due to a string handling bug the check for already backends always found nothing, leading to multiple conflicting backends loaded on some systems (for example, those with libical0 and libical1 installed). If that happened, the backend became unusable. * ical: workaround for libical 1.0 builtin timezone change libical 1.0 started to return VTIMEZONE definitions with multiple absolute transition times instead of RRULEs. This causes problems when exchanging data with peers (see https://sourceforge.net/p/freeassociation/bugs/95/). In SyncEvolution, this affected sending an event using New Zealand time in vCalendar 1.0 format to a phone, because the internal, out-dated definition of the time zone in libsynthesis was used as fallback when loading RRULE-based timezone definitions from libical failed (see "[SyncEvolution] Some events showing wrong time on phone"). It might also affect exchanging data with CalDAV peers (not tested). The workaround is to include the original code from libical. * dbus-session.sh: create XDG_RUNTIME_DIR More recent distros (for example, Ubuntu Saucy) rely on XDG_RUNTIME_DIR. Each time dbus-session.sh runs, it must ensure that the runtime dir exists and is empty. This was a problem when trying to run activesyncd + SyncEvolution on a headless Ubuntu Saucy server (see FDO #76273). * Akonadi: support KDE Notes, enhanced "database" check The KDE Notes resources store items under a different MIME type than the one used in AKonadi (see "[Kde-pim] note format"). SyncEvolution use the same type as Akonadi and thus did not find existing KDE Notes resources. To support both while KDE and Akonadi transition to the same type, SyncEvolution now looks for notes resources using both MIME types and accepts both kinds of items when reading. When writing, SyncEvolution picks the MIME type that is supported by the resource, which hopefully avoids confusing the KDE app using the resource (untested). As a positive side effect, the "database" value used for opening a resource is now checked more thoroughly. Non-existent resources and the type mismatches like pointing a "kde-contacts" backend to a calendar resource are now detected early. * Akonadi: ensure that UID is set (FDO #74342) Akonadi resources do not enforce iCalendar 2.0 semantic like "each VEVENT must have a UID" (see "[Kde-pim] iCalendar semantic"). When receiving an event from a peer which itself does not enforce that semantic (Funambol, vCalendar 1.0 based phones), then we need to generate a UID, otherwise KOrganizer will ignore the imported event. * Akonadi: avoid threading problem in HTTP server mode (FDO #75672) When used as storage in a server, Akonadi got called in a background thread that gets created to handle slow initialization of sources and preventing ensuing timeouts in HTTP clients (probably not needed for Akonadi itself, but may still be useful when combining it with other sources). Akonadi cannot be used like that, leading to false "Akonadi not running" errors or (if one got past that check) failing item operations. * autotools: Add QtCore include path to KDEPIM_CFLAGS (FDO #75670) This fixes an issue where configure fails to find Akonadi when test programs do not compile because QString is not found. * Enhanced testing again: faster execution, less false negatives under load. Re-enabled testing of Akonadi. SyncEvolution 1.3.2 -> 1.4, 16.02.2014 ====================================== The 1.4 relase of SyncEvolution is the first stable version with the in-vehicle infotainment (IVI) PIM Manager included. GENIVI Diagnostic Log and Trace (DLT) is also supported. For more information about this aspect of SyncEvolution, see the PBAP and PIM entries in the 1.3.99.x release notes and https://syncevolution.org/blogs/pohly/2013/pim-its-all-about-contacts The biggest change for normal Linux users is Google CalDAV/CardDAV authentication with OAuth2. These are the open protocol that Google currently supports and thus the recommended way of syncing with Google, replacing ActiveSync and SyncML (both no longer available to all Google customers). Support for Google CardDAV is new. Like Evolution, SyncEvolution does not yet support some of the advanced features of the server, in particular custom labels for phone numbers, emails and addresses. Likewise, some client properties are not supported by the server: CALURI, CATEGORIES, FBURL, GEO and ROLE are not supported. Of ORG, only the first two components are supported. Currently, properties not supported by one side get lost in a full roundtrip sync. SyncEvolution depends on external components for OAuth2. It can be compiled to use gSSO [1] or GNOME Online Accounts [2]. The latter is enabled in binaries from syncevolution.org. GNOME Online Accounts >= 3.10 works out of the box for CalDAV and CardDAV. 3.8 is guaranteed to work for CardDAV and may also work for CalDAV, if the Linux distribution ships a patched version (like Debian Wheezy does). If it does not, then GNOME Online Accounts 3.8 binary can be patched to also support CalDAV, see [2]. Anything older than 3.8 does not work. Support for Ubuntu Online Accounts is available when compiling from source. For setup instructions see these READMEs. [1] https://01.org/gsso and http://cgit.freedesktop.org/SyncEvolution/syncevolution/plain/src/backends/signon/README [2] http://cgit.freedesktop.org/SyncEvolution/syncevolution/plain/src/backends/goa/README Binary packages of 1.4 on syncevolution.org have enhanced support for recent distros. They now work with EDS >= 3.6 *and* < 3.6. Distros with libical1 like Ubuntu Saucy are also supported. The HTTP server became better at handling message resends when the server is slow with processing a message. The server is able to keep a sync session alive while loading the initial data set by sending acknowledgement replies before the client times out. Some issues in CalDAV, WebDAV and SyncML were fixed. Graham R. Cobb contributed several patches for enhancing ActiveSync support and making it work with Exchange 2010. Guido Günther provided some patches addressing problems when compiling SyncEvolution for Maemo. Details: * D-Bus server: support DLT (FDO #66769) Diagnostic Log and Trace (DLT) manages a sequence of log messages, with remote controllable level of detail. SyncEvolution optionally (can be chosen at compile time and again at runtime) uses DLT instead of its own syncevolution-log.html files. See README-DLT.rst for more information. To use the feature, configure SyncEvolution with "--enable-dbus-server=--dlt --no-syslog" * D-Bus server: fix abort when mixing auto-sync and manual operations (FDO #73562) When enabling auto-sync for a config and then accessing or syncing the config manually via the command line tool, the server would abort at the time when the auto-sync was originally scheduled. * D-Bus server: accept WBXML with charset in incoming connections A user reported via email that the Nokia 515 sends 'application/vnd.syncml+wbxml; charset=UTF-8' as type of its messages this tripped up the syncevo-http-server, leading to: [ERROR] syncevo-dbus-server: /org/syncevolution/Server: message type 'application/vnd.syncml+wbxml; charset=UTF-8' not supported for starting a sync * D-Bus server: command line options for controlling output and startup The system log is used by default now. New command line options can be used to change this: -d, --duration=seconds/'unlimited' Shut down automatically when idle for this duration (default 300 seconds) -v, --verbosity=level Choose amount of output, 0 = no output, 1 = errors, 2 = info, 3 = debug; default is 1. --dbus-verbosity=level Choose amount of output via D-Bus signals, 0 = no output, 1 = errors, 2 = info, 3 = debug; default is 2. -o, --stdout Enable printing to stdout (result of operations) and stderr (errors/info/debug). -s, --no-syslog Disable printing to syslog. -p, --start-pim Activate the PIM Manager (= unified address book) immediately. * D-Bus: missing out parameters in D-Bus introspection XML (FDO #57292) The problem was in the C++ D-Bus binding. If the method that gets bound to D-Bus returns a value, that value was ignored in the signature: int foo() => no out parameter It works when the method was declared as having a retval: void foo (int &result) => integer out parameter This problem existed for both the libdbus and the GIO D-Bus bindings. In SyncEvolution it affected methods like GetVersions(). * D-Bus server: avoid progress outside of 0-100% range For example in the new TestLocalCache.testItemDelete100, the percentage value in the ProgressChanged signal become larger than 100 and then revert to 100 at the end of the sync. Seems the underlying calculation is faulty or simply inaccurate. This is not fixed. Instead the result is just clipped to the valid range. * sync: less verbose output, shorter runtime For each incoming change, one INFO line with "received x[/out of y]" was printed, immediately followed by another line with total counts "added x, updated y, removed z". For each outgoing change, a "sent x[/out of y]" was printed. In addition, these changes were forwarded to the D-Bus server where a "percent complete" was calculated and broadcasted to clients. All of that caused a very high overhead for every single change, even if the actual logging was off. The syncevo-dbus-server was constantly consuming CPU time during a sync when it should have been mostly idle. To avoid this overhead, the updated received/sent numbers that come from the Synthesis engine are now cached and only processed when done with a SyncML message or some other event happens (whatever happens first). To keep the implementation simple, the "added x, updated y, removed z" information is ignored completely and no longer appears in the output. * command line: implement --create/remove-database Creating a database is only possible with a chosen name. The UID is chosen automatically by the storage. Only implemented in the EDS backend. * command line: execute --export and --print-items while the source is still reading Instead of reading all item IDs, then iterating over them, process each new ID as soon as it is available. With sources that support incremental reading (only the PBAP source at the moment) that provides output sooner and is a bit more memory efficient. * command line: recover from slow sync with new sync modes The error message for an unexpected slow sync still mentioned the old and obsolete "refresh-from-client/server" sync modes. Better mention "refresh-from-local/remote". * command line: show backend error when listing databases fails The command line swallowed errors thrown by the backend while listing databases. Instead it just showed ": backend failed". The goal was to not distract users who accidentally access a non-functional backend. But the result is that operations like --configure or --print-databases could fail without giving the user any hint about the root cause of the issue. Now the error explanation in all its gory details is included. For example, not having activesyncd running leads to: INFO] eas_contact: backend failed: fetching folder list: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.meego.activesyncd was not provided by any .service files And running activesyncd without the necessary gconf keys shows up as: [INFO] eas_contact: backend failed: fetching folder list: GDBus.Error:org.meego.activesyncd.Error.AccountNotFound: Failed to find account [syncevolution@lists.intel.com] * password handling: fix usage of GNOME Keyring and KWallet (FDO #66110) When clients like the GTK sync-ui stored a password, it was always stored as plain text in the config.ini file by the syncevo-dbus-server. The necessary code for redirecting the password storage in a keyring (GNOME or KWallet) simply wasn't called in that case. The command line tool, even when using the D-Bus server to run the operation, had the necessary code active and thus was not affected. Now all SyncEvolution components use the same default: use safe password storage if either GNOME Keyring or KWallet were enabled during compilation, don't use it if not. Fixing this revealed other problems, like not being able to store certain passwords that lacked the necessary lookup criteria (like syncURL and/or username). To address this, the lookup criteria where extended and a new check was added to avoid accidentally removing other passwords. As a result, it may be possible that SyncEvolution no longer finds passwords that were stored with older versions of SyncEvolution. In such a case the passwords must be set again. * GNOME: clean up keyring access and require libgnome-keyring >= 2.20 The updated error messages now always include information about the password and libgnome-keyring error texts. A workaround is used for the "Error communicating with gnome-keyring-daemon" problem that started to appear fairly frequently in the automated testing once the keyring was actually used. The problem shows up with some additional debug messages: Gkr: received an invalid, unencryptable, or non-utf8 secret Gkr: call to daemon returned an invalid response: (null).(null)() It seems that sometimes setting up a session with GNOME keyring fails such that all further communication leads to decoding problem. There is an internal method to reset the session, but it cannot be called directly. As a workaround, fake the death of the GNOME keyring daemon and thus trigger a reconnect when retrying the GNOME keyring access. This is done by sending a D-Bus message, which will also affect other clients of GNOME keyring, but hopefully without user-visible effects. * config: enhanced password handling It is possible to configure a plain username/password combination once in SyncEvolution and then use references to it in other configurations, instead of having to set (and update) the credentials in different places. This is useful in particular with WebDAV, where credentials had to be repeated several times (target config, in each database when used as part of SyncML) or when using a service which requires several configs (Google via SyncML and CalDAV). To use this, create a sync config for a normal peer or a dedicated config just for the credentials, with "username/password/syncURL" set. The "syncURL" must be set to something identifying the peer if GNOME Keyring is used for the password storage. Then set "username", "databaseUser" and "proxyUser" properties to "id:" and all read and write access to those properties will be redirected by SyncEvolution into that other configuration. This even works in the GTK UI. For user names which contain colons, the new "user:" format must be used. Strings without colons are assumed to be normal user names, so most old configurations should continue to work. * signon: new backend using libgsignond-glib + libaccounts-glib The code works with gSSO (https://01.org/gsso) and Ubuntu Online Accounts. * GOA: get OAuth2 tokens out of GNOME Online Accounts "username = goa:..." selects an account in GOA and retrieves the OAuth2 token from that. * WebDAV: support OAuth2 If given an authentication configuration which can handle OAuth2, then OAuth2 is used instead of plain username/password authentication. * WebDAV: support Google CardDAV, break Yahoo Google CardDAV has one peculiarity: it renames new contacts during PUT without returning the new path to the client. See also http://lists.calconnect.org/pipermail/caldeveloper-l/2013-July/000524.html SyncEvolution already had a workaround for that (PROPGET on old path, extract new path from response) which happened to work. This workaround was originally added for Yahoo, which sometimes merges contacts into existing ones. In contrast to Yahoo, Google really seems to create new items. Without some server specific hacks, the client cannot tell what happened. Because Google is currently supported and Yahoo is not, let's change the hard-coded behavior to "renamed items are new". * WebDAV: started testing with owndrive.com = OwnCloud * WebDAV: avoid segfault during collection lookup Avoid referencing pathProps->second when the set of paths that PROPFINDs returns is empty. Apparently this can happen in combination with Calypso. * CalDAV: more workarounds for Google CalDAV + unique IDs Google became even more strict about checking REV. Tests which reused a UID after deleting the original item started to fail sometime since middle of December 2012. * CalDAV: work around Google server regression (undeclared namespace prefix in XML) Google CalDAV for a while (December 2012 till January 2013) sent invalid XML back when asked to include CardDAV properties in a PROPFIND. This got rejected in the XML parser, which prevents syncing calendar data: Neon error code 1: XML parse error at line 55: undeclared namespace prefix In the meantime Google fixed the issue in response to a bug report via email. But the workaround, only asking for the properties which are really needed, still makes sense and thus is kept. * WebDAV: auto-discovery fix With Google Contact + CardDAV the auto-discovery failed after finding the default address book, without reporting that result. * WebDAV: don't send Basic Auth via http proactively (FDO #57248) Sending basic authentication headers via http is insecure. Only do it proactively when the connection is encrypted and thus protects the information or when the server explicitly asks for it. * file backend: sub-second mod time stamps Change tracking in the file backend used to be based on the modification time in seconds. When running many syncs quickly (as in testing), that can lead to changes not being detected when they happen within a second. Now the file backend also includes the sub-second part of the modification time stamp, if available. This change is relevant when upgrading SyncEvolution: most of the items will be considered "updated" once during the first sync after the upgrade (or a downgrade) because the revision strings get calculated differently. * GTK UI: fixed two crashes - running a sync with no service selected and a 64 bit pointer problem recently discovered by Tino Keitel when compiling the Debian package with -fPIE. * packaging: compatible with EDS up to and including 3.10 and both libical.so.0 and libical.so.1 The binary packages now contain different versions of syncecal.so and syncebooks.so to cover different combinations of EDS and libical. * libical: compatibiliy mode for libical.so.0 and libical.so.1 libical 1.0 broke the ABI, leading to libical.so.1. The only relevant change for SyncEvolution is the renumbering of ICAL_*_PROPERTY enum values. We can adapt to that change at runtime, which allows us to compile once with libical.so.0, then patch executables or use dynamic loading to run with the more recent libical.so.1 if we add 1 to the known constants. * packaging: fix rpm (FDO #73347) After installing the syncevolution.org rpm on OpenSUSE, SyncEvolution was not starting because its shared libraries were not found unless "ldconfig" was called manually. Now the package does that automatically. * packaging: fix description The syncevolution-bundle description of both rpm and deb packagesaccidentally used the same description as syncevolution-evolution. * glib: fix double-free of source tags glib 2.39.0 (aka GNOME 3.10) as found in Ubuntu Trusty introduces warnings when g_source_remove() is passed an unknown tag. SyncEvolution did this in two cases: in both, the source callback returned false and thus caused the source to be removed by the caller. In that case, the explicit g_source_remove() is redundant and must be avoided. Such a call is faulty and might accidentally remove a new source with the same tag (unlikely though, given that tags seem to get assigned incrementally). The only noticable effect were additional error messages with different numbers: [ERROR] GLib: Source ID 9 was not found when attempting to remove it * EDS: fix compile problem with boost and EDS > 3.36 This fixes the following problem, seen with Boost 1.53.0 on altlinux when compiling for EDS >= 3.6: /usr/include/boost/smart_ptr/shared_ptr.hpp: In instantiation of 'typename boost::detail::sp_array_access::type boost::shared_ptr::operator[](std::ptrdiff_t) const [with T = char*; typename boost::detail::sp_array_access::type = void; std::ptrdiff_t = long int]': src/backends/evolution/EvolutionSyncSource.cpp:163:38: required from here /usr/include/boost/smart_ptr/shared_ptr.hpp:663:22: error: return-statement with a value, in function returning 'void' [-fpermissive] make[2]: *** [src/backends/evolution/src_backends_evolution_syncecal_la-EvolutionSyncSource.lo] * EDS contacts: avoid unnecessary DB writes during slow sync Traditionally, contacts were modified shortly before writing into EDS to match with Evolution expectations (must have N, only one CELL TEL, VOICE flag must be set). During a slow sync, the engine compare the modified contacts with the unmodified, incoming one. This led to mismatches and/or merge operations which end up not changing anything in the DB because the only difference would be removed again before writing. * EDS contacts: read-ahead cache Performance is improved by requesting multiple contacts at once and overlapping reading with processing. On a fast system (SSD, CPU fast enough to not be the limiting factor), testpim.py's testSync takes 8 seconds for a "match" sync where 1000 contacts get loaded and compared against the same set of contacts. Read-ahead with only 1 contact per query speeds that up to 6.7s due to overlapping IO and processing. Read-ahead with the default 50 contacts per query takes 5.5s. It does not get much faster with larger queries. * PBAP: add support for obexd 0.47, 0.48 and Bluez 5 obexd 0.48 is almost the same as obexd 0.47, except that it dropped the SetFilter and SetFormat methods in favor of passing a Bluex 5-style filter parameter to PullAll. * PBAP: various enhancements for efficient caching of contacts * HTTP server: handle message resends If a client gave up waiting for the server's response and resent its message while the server was still processing the message, syncing failed with "protocol error: already processing a message" raised by the syncevo-dbus-server because it wasn't prepared to handle that situation. The right place to handle this is inside the syncevo-http-server, because it depends on the protocol (HTTP in this case) whether resending is valid or not. It handles that now by tracking the message that is currently in processing and matching it against each new message. If it matches, the new request replaces the obsolete one without sending the message again to syncevo-dbus-server. When syncevo-dbus-server replies to the old message, the reply is used to finish the newer request. * engine: prevent timeouts in HTTP server mode HTTP SyncML clients give up after a certain timeout (SyncEvolution after RetryDuration = 5 minutes by default, Nokia e51 after 15 minutes) when the server fails to respond. This can happen with SyncEvolution as server when it uses a slow storage with many items, for example via WebDAV. In the case of slow session startup, multithreading is now used to run the storage initializing in parallel to sending regular "keep-alive" SyncML replies to the client. By default, these replies are sent every 2 minutes. This can be configured with another extensions of the SyncMLVersion property: SyncMLVersion = REQUESTMAXTIME=5m Other modes do not use multithreading by default, but it can be enabled by setting REQUESTMAXTIME explicitly. It can be disabled by setting the time to zero. The new feature depends on a libsynthesis with multithreading enabled and glib >= 2.32.0, which is necessary to make SyncEvolution itself thread-safe. With an older glib, multithreading is disabled, but can be enabled as a stop-gap measure by setting REQUESTMAXTIME explicitly. * Various testing and stability enhancements. SyncEvolution had to be made thread-safe for the HTTP timeout prevention. * Nokia: always add TYPE=INTERNET to EMAIL (FDO #61784) Without the explicit TYPE=INTERNET, email addresses sent to a Nokia e51 were not shown by the phone and even got lost eventually (when syncing back). This commit ensures that the type is set for all emails sent to any Nokia phone, because there may be other phones which need it and phones which don't, shouldn't mind. This was spot-checked with a N97 mini, which works fine with and without the INTERNET type. This behavior can be disabled again for specific Nokia phones by adding a remote rule which sets the addInternetEmail session variable to FALSE again. Non-Nokia phones can enable the feature in a similar way, by setting the variable to TRUE. * SyncML: config option for broken peers Some peers have problems with meta data (CtCap, old Nokia phones) and the sync mode extensions required for advertising the restart capability (Oracle Beehive). The default in SyncEvolution is to advertise the capability, so manual configuration is necessary when working with a peer that fails in that mode. Because the problem occurs when SyncEvolution contacts the peers before it gets the device information from the peer, dynamic rules based on the peer identifiers cannot be used. Instead the local config must already disable these extra features in advance. The "SyncMLVersion" property gets extended for this. Instead of just "SyncMLVersion = 1.0" (as before) it now becomes possible to say "SyncMLVersion = 1.0, noctcap, norestart". "noctcap" disables sending CtCap. "norestart" disables the sync mode extensions and thus doing multiple sync cycles in the same session (used between SyncEvolution instances in some cases to get client and server into sync in one session). Both keywords are case-insensitive. There's no error checking for typos, so beware! The "SyncMLVersion" property was chosen because it was already in use for configuring SyncML compatibility aspects and adding a new property would have been harder. * ActiveSync: added support for specifying folder names Previously, the database field was interpreted as a Collection ID. This adds logic to allow the database to be interpreted as a folder path. The logic is: 1) If the database is an empty string, pass it through (this is the most common case as it is interpreted as "use the default folder for the source type"). 2) If the database matches a Collection ID, use the ID (this is the same as the previous behaviour). 3) If the database matches a folder path name, with an optional leading "/", use the Collection ID for the matching folder. 4) Otherwise, force a FolderSync to get the latest folder changes from the server and repeat steps 2 and 3 5) If still no match, throw an error. * ActiveSync: support for listing databases Now --print-databases scans folders on the ActiveSync server and shows suitable folders for the ActiveSync backends instead of the previous, hard-coded help text. Invoking --print-databases can be used as a workaround for "SyncFolder error: Invalid synchronization key" errors. A better solution would be to do that automatically, but there was no time to implement that. See FDO #61869 and "[SyncEvolution] Activesync server losing state" http://thread.gmane.org/gmane.comp.mobile.syncevolution/4295 * SyncML: workarounds for broken peers Some peers have problems with meta data (CtCap, old Nokia phones) and the sync mode extensions required for advertising the restart capability (Oracle Beehive). Because the problem occurs when SyncEvolution contacts the peers before it gets the device information from the peer, dynamic rules based on the peer identifiers cannot be used. Instead the local config must already disable these extra features in advance. The "SyncMLVersion" property gets extended for this. Instead of just "SyncMLVersion = 1.0" (as before) it now becomes possible to say "SyncMLVersion = 1.0, noctcap, norestart". "noctcap" disables sending CtCap. "norestart" disables the sync mode extensions and thus doing multiple sync cycles in the same session (used between SyncEvolution instances in some cases to get client and server into sync in one session). Both keywords are case-insensitive. There's no error checking for typos, so beware! The "SyncMLVersion" property was chosen because it was already in use for configuring SyncML compatibility aspects and adding a new property would have been harder. * engine: local cache sync mode This patch introduces support for true one-way syncing ("caching"): the local datastore is meant to be an exact copy of the data on the remote side. The assumption is that no modifications are ever made locally outside of syncing. This is different from one-way sync modes, which allows local changes and only temporarily disables sending them to the remote side. Another goal of the new mode is to avoid data writes as much as possible. This new mode only works on the server side of a sync, where the engine has enough control over the data flow. Setting "sync" to: - "local-cache-incremental" will do an incremental sync (if possible) or a slow sync (otherwise). This is usually the right mode to use, and thus has "local-cache" as alias. - "local-cache-slow" will always do a slow sync. Useful for debugging or after (accidentally) making changes on the local side. An incremental sync will ignore such changes because they are not meant to happen, aren't checked for to improve performance and thus will leave client and server out-of-sync! Both modes are recorded in the sync report of the local side. The target side is the client and records the normal "two-way" or "slow" sync modes. With the current SyncEvolution contact field list, first, middle and last name are used to find matches for contacts. For events, tasks and memos, time, summary and description are used. * Minor memory leak fix when using GDBus GIO: GDBusMethodInfo Also depends on a glib fix, see https://bugzilla.gnome.org/show_bug.cgi?id=695376 * build fixes Avoid -lrt in make dependencies. Add missing pcre libs to syncevo-dbus-server. sqlite backend needs "#include " (patch from Mario Kicherer). * autotools: fix temp file vulnerability during compilation (CVE-2014-1639) We must use the temporary file that was created for us securily, not a temp file named after that file. This caused a temp file vulnerability and the real temporary files were not deleted by the script. * workarounds for warnings from g++ 4.5 Upgrading from releases <= 1.3.99.4: If the value of "username/databaseUser/proxyUser" contains a colon, the "user:" prefix must be added to the value, to continue treating it like a plain user name and not some reference to an unknown identity provider (like "id:", "goa:", "signon:", etc.). The lookup of passwords in GNOME Keyring was updated slightly in 1.3.99.5. It may be necessary to set passwords anew if the old one is no longer found. Upgrading from release 1.2.x: The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution --configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the "refresh-from-server" sync mode to avoid getting throttled with 417 'retry later' errors. The same must be added to existing configs manually: syncevolution --configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2: Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. SyncEvolution 1.3.99.7 -> 1.4, 16.02.2014 ========================================= Compared to the pre-release, 1.4 mostly just enhanced the testing. Compatibility with GNOME 3.10 and a glib-related issue that existed almost forever without causing obvious problems were fixed. syncevolution.org binaries now finally work with distros using libical.so.1 (for example, Ubuntu Saucy and Trusty). Details: * autotools: fix temp file vulnerability during compilation (CVE-2014-1639) We must use the temporary file that was created for us securily, not a temp file named after that file. This caused a temp file vulnerability and the real temporary files were not deleted by the script. * glib: fix double-free of source tags glib 2.39.0 (aka GNOME 3.10) as found in Ubuntu Trusty introduces warnings when g_source_remove() is passed an unknown tag. SyncEvolution did this in two cases: in both, the source callback returned false and thus caused the source to be removed by the caller. In that case, the explicit g_source_remove() is redundant and must be avoided. Such a call is faulty and might accidentally remove a new source with the same tag (unlikely though, given that tags seem to get assigned incrementally). The only noticable effect were additional error messages with different numbers: [ERROR] GLib: Source ID 9 was not found when attempting to remove it * libical: compatibiliy mode for libical.so.0 and libical.so.1 libical 1.0 broke the ABI, leading to libical.so.1. The only relevant change for SyncEvolution is the renumbering of ICAL_*_PROPERTY enum values. We can adapt to that change at runtime, which allows us to compile once with libical.so.0, then patch executables or use dynamic loading to run with the more recent libical.so.1 if we add 1 to the known constants. SyncEvolution 1.3.99.7, 22.01.2014 ================================== Final release candidate for 1.4. No further changes planned unless new problems are found. Details: * SSO: support Ubuntu Online Accounts When compiling from source on recent Ubuntu it becomes possible to use Ubuntu Online Accounts for authenticating against Google's CalDAV and CardDAV servers. * D-Bus server: fix abort when mixing auto-sync and manual operations (FDO #73562) When enabling auto-sync for a config and then accessing or syncing the config manually via the command line tool, the server would abort at the time when the auto-sync was originally scheduled. * D-Bus server: accept WBXML with charset in incoming connections A user reported via email that the Nokia 515 sends 'application/vnd.syncml+wbxml; charset=UTF-8' as type of its messages this tripped up the syncevo-http-server, leading to: [ERROR] syncevo-dbus-server: /org/syncevolution/Server: message type 'application/vnd.syncml+wbxml; charset=UT We need to strip the '; charset=UTF-8' suffix also when checking for WBXML. * packaging: compatible with EDS up to and including 3.10 The packages now contain three versions of syncecal.so: - one for EDS < 3.6 - one for EDS >= 3.6 < 3.10 - one for EDS >= 3.10 with the libecal-1.2 soname patched The third flavor became necessary because EDS 3.10 accidentally changed the soname. The API and ABI actually is the same. Package meta-data was fixed to reflect the extended range of compatible EDS libraries, so syncevolution-evolution can be installed again with recent EDS. * packaging: update syncevolution-kde dependencies kdebase-runtime became kde-runtime in Debian Wheezy. Accept both as prerequisite of syncevolution-kde to allow installation on newer distros without pulling in the transitional kdebase-runtime package. * packaging: fix rpm (FDO #73347) After installing the syncevolution.org rpm on OpenSUSE, SyncEvolution was not starting because its shared libraries were not found unless "ldconfig" was called manually. Now the package does that automatically. * packaging: fix description The syncevolution-bundle description of both rpm and deb packagesaccidentally used the same description as syncevolution-evolution. * test improvements, integration of cppcheck and clang's scan-build SyncEvolution 1.3.99.6, 04.12.2013 ================================== This update focuses on SyncEvolution in IVI again. It adds support for GENIVI Diagnostic Log and Trace (DLT) and enhances searching in the unified address book. The biggest change for normal Linux desktop users is enhanced support for recent distros. Binaries on syncevolution.org now work with EDS >= 3.6 *and* < 3.6. Distros with libical1 like Ubuntu Saucy are also supported. Automated testing was updated to cover these newer platforms more thoroughly. Details: * GNOME Online Accounts: fix D-Bus problem in syncevolution.org binaries Support was included in syncevolution.org binaries, but was not tested and did not actually work due to some issue accessing the D-Bus session. * libsynthesis: partial fix batching of items The batching of contact writes introduced with SyncEvolution 1.3.99.4 caused problems with non-SyncEvolution SyncML peers when syncing contacts stored in EDS >= 3.6. EDS < 3.6 was not affected. That part is fixed. However, even in SyncEvolution<->SyncEvolution syncs another crash was found. This will require more investigation. Clearly the feature is not ready yet for general sync, so for now it is disabled by default and only enabled in the simpler PBAP sync. * libsynthesis: avoid redundant (and sometimes slow) getaddrbyname() (FDO #70771) The network lookup of the hostname can be slow (10 second delay when not connected) and shouldn't be necessary anyway, so disable it. * PIM Manager: case-insensitive and transliterated search (FDO #56524) * PIM: accent-insensitive and transliterated search (FDO #56524) Accent-insensitive search ignores accents, using the same code as in EDS. Transliterated search ignores foreign scripts by transliterating search term and contact properties to Latin first. That one is using ICU directly in the same way as EDS, but doesn't use the EDS ETransliterator class to avoid extra string copying. This commit changes the default behavior such that searching is by default most permissive (case- and accent-insensitive, does transliteration). Flags exist to restore more restrictive matching. * PIM: relax phone number matching Previously, the current default country was used to turn phone numbers without an explicit country code into full E164 numbers, which then had to match the search term when doing a caller ID lookup. This was inconsistent with EDS, where a weaker EQUALS_NATIONAL_PHONE_NUMBER was done. The difference is that a comparison between a number with country code matches one without if the national number of the same, regardless of the current default country. This is better because it reduces the influence of the hard to guess default country on matching. Another advantage of this change is the lower memory consumption and faster comparison, because strings are now stored in 4 + 8 byte numbers instead of strings of varying length. * PIM: fix incorrect write into pim-manager.ini (FDO #70772) Removing a peer accidentally wrote the updated list of active address books into the "sort" property of pim-manager.ini, which then prevented starting the PIM Manager. * PIM: ignore broken sort order in config (FDO #70772) Failure to set the sort order from pim-manager.ini should not prevent the startup of the PIM Manager because the client cannot really diagnose and fix the problem. It is better to try again with the default sort order. * PIM: adapt to locale changes at runtime (FDO #66618) Listen to signals from localed D-Bus system service and update all internal state which depends on the current locale. This state includes: - pre-computed data in all loaded contacts - filtering (for example, case sensitivity is locale dependent) - the sort order This feature can be controlled by setting the SYNCEVOLUTION_LOCALED env variable: - "session" - use a localed instance on the D-Bus session bus instead of the system instance. This was originally meant for testing, but might also be useful for per-user setting changes. - "none" - disables the feature * PIM: fix sync.py + multiple peers Due to overwriting a variable, configuring multiple different peers did not work. * D-Bus server: support DLT (FDO #66769) Diagnostic Log and Trace (DLT) manages a sequence of log messages, with remote controllable level of detail. SyncEvolution optionally (can be chosen at compile time and again at runtime) uses DLT instead of its own syncevolution-log.html files. See README-DLT.rst for more information. To use the feature, configure SyncEvolution with "--enable-dbus-server=--dlt --no-syslog" * EDS: enhanced compatibility mode SyncEvolution compiled for EDS < 3.6 can now also load EDS backends compiled for EDS >= 3.6. The packaging for syncevolution.org uses that to bundle EDS backends compiled on different distros in the same package. * EDS: SYNCEVOLUTION_EBOOK_QUERY env variable Setting the SYNCEVOLUTION_EBOOK_QUERY env variable to a valid EBook query string limits the results to contacts matching that query. Useful only in combination with --print-items or --export. Only implemented for EDS >= 3.6. * EDS: fix compile problem with boost and EDS > 3.36 This fixes the following problem, seen with Boost 1.53.0 on altlinux when compiling for EDS >= 3.6: /usr/include/boost/smart_ptr/shared_ptr.hpp: In instantiation of 'typename boost::detail::sp_array_access::type boost::shared_ptr::operator[](std::ptrdiff_t) const [with T = char*; typename boost::detail::sp_array_access::type = void; std::ptrdiff_t = long int]': src/backends/evolution/EvolutionSyncSource.cpp:163:38: required from here /usr/include/boost/smart_ptr/shared_ptr.hpp:663:22: error: return-statement with a value, in function returning 'void' [-fpermissive] make[2]: *** [src/backends/evolution/src_backends_evolution_syncecal_la-EvolutionSyncSource.lo] * PBAP: add support for obexd 0.48 obexd 0.48 is almost the same as obexd 0.47, except that it dropped the SetFilter and SetFormat methods in favor of passing a Bluex 5-style filter parameter to PullAll. SyncEvolution now supports 4, in words, four different obexd APIs. Sigh. This feature was originally announced for SyncEvolution 1.3.99.5, but not actually included in the code yet. SyncEvolution 1.3.99.5, 01.10.2013 ================================== SyncEvolution now supports Google CalDAV/CardDAV with OAuth2 authentication. These are the open protocol that Google currently supports and thus the recommended way of syncing with Google, replacing ActiveSync and SyncML (both no longer available to all Google customers). Support for Google CardDAV is new. Because of a vCard encoding issue on the server side, spaces in long notes may get removed. Like Evolution, SyncEvolution does not yet support some of the advanced features of the server, in particular custom labels for phone numbers, emails and addresses. Likewise, some client properties are not supported by the server: CALURI, CATEGORIES, FBURL, GEO and ROLE are not supported. Of ORG, only the first two components are supported. Currently, properties not supported by one side get lost in a full roundtrip sync. SyncEvolution depends on external components for OAuth2. It can be compiled to use gSSO [1] or GNOME Online Accounts. GNOME Online Accounts >= 3.10 works out of the box for CalDAV and CardDAV, 3.8 only for CardDAV (but the GNOME Online Accounts binary can be patched to also support CalDAV, see [2]), anything older than 3.8 does not work. Support for Ubuntu Online Accounts should not be hard to add, but is not available yet [3]. [1] https://01.org/gsso and http://cgit.freedesktop.org/SyncEvolution/syncevolution/plain/src/backends/signon/README [2] http://cgit.freedesktop.org/SyncEvolution/syncevolution/plain/src/backends/goa/README [3] http://thread.gmane.org/gmane.comp.mobile.syncevolution/4353/focus=4490 Details: * GTK UI: fixed two crashes - running a sync with no service selected and a 64 bit pointer problem recently discovered by Tino Keitel when compiling the Debian package with -fPIE. * password handling: fix usage of GNOME Keyring and KWallet (FDO #66110) When clients like the GTK sync-ui stored a password, it was always stored as plain text in the config.ini file by the syncevo-dbus-server. The necessary code for redirecting the password storage in a keyring (GNOME or KWallet) simply wasn't called in that case. The command line tool, even when using the D-Bus server to run the operation, had the necessary code active and thus was not affected. Now all SyncEvolution components use the same default: use safe password storage if either GNOME Keyring or KWallet were enabled during compilation, don't use it if not. Fixing this revealed other problems, like not being able to store certain passwords that lacked the necessary lookup criteria (like syncURL and/or username). To address this, the lookup criteria where extended and a new check was added to avoid accidentally removing other passwords. As a result, it may be possible that SyncEvolution no longer finds passwords that were stored with older versions of SyncEvolution. In such a case the passwords must be set again. * GNOME: clean up keyring access and require libgnome-keyring >= 2.20 The updated error messages now always include information about the password and libgnome-keyring error texts. A workaround is used for the "Error communicating with gnome-keyring-daemon" problem that started to appear fairly frequently in the automated testing once the keyring was actually used. The problem shows up with some additional debug messages: Gkr: received an invalid, unencryptable, or non-utf8 secret Gkr: call to daemon returned an invalid response: (null).(null)() It seems that sometimes setting up a session with GNOME keyring fails such that all further communication leads to decoding problem. There is an internal method to reset the session, but it cannot be called directly. As a workaround, fake the death of the GNOME keyring daemon and thus trigger a reconnect when retrying the GNOME keyring access. This is done by sending a D-Bus message, which will also affect other clients of GNOME keyring, but hopefully without user-visible effects. * config: enhanced password handling It is possible to configure a plain username/password combination once in SyncEvolution and then use references to it in other configurations, instead of having to set (and update) the credentials in different places. This is useful in particular with WebDAV, where credentials had to be repeated several times (target config, in each database when used as part of SyncML) or when using a service which requires several configs (Google via SyncML and CalDAV). To use this, create a sync config for a normal peer or a dedicated config just for the credentials, with "username/password/syncURL" set. The "syncURL" must be set to something identifying the peer if GNOME Keyring is used for the password storage. Then set "username", "databaseUser" and "proxyUser" properties to "id:" and all read and write access to those properties will be redirected by SyncEvolution into that other configuration. This even works in the GTK UI. For user names which contain colons, the new "user:" format must be used. Strings without colons are assumed to be normal user names, so most old configurations should continue to work. * signon: new backend using libgsignond-glib + libaccounts-glib The code works with gSSO (https://01.org/gsso). With some tweaks to the configure check and some ifdefs it probably could be made to work with Ubuntu Online Accounts. The code depends on an account accessible via libaccounts-glib which has a provider and and (optionally) services enabled for that provider. It is not necessary that the account already has a signon identity ID, the backend will create that for the provider (and thus shared between all services) if necessary. Therefore it is possible to use the ag-tool to create and enable the account and services. Provider and service templates are in the next commit. * WebDAV: support OAuth2 If given an authentication configuration which can handle OAuth2, then OAuth2 is used instead of plain username/password authentication. * WebDAV: support Google CardDAV, break Yahoo Google CardDAV has one peculiarity: it renames new contacts during PUT without returning the new path to the client. See also http://lists.calconnect.org/pipermail/caldeveloper-l/2013-July/000524.html SyncEvolution already had a workaround for that (PROPGET on old path, extract new path from response) which happened to work. This workaround was originally added for Yahoo, which sometimes merges contacts into existing ones. In contrast to Yahoo, Google really seems to create new items. Without some server specific hacks, the client cannot tell what happened. Because Google is currently supported and Yahoo is not, let's change the hard-coded behavior to "renamed items are new". * WebDAV: started testing with owndrive.com = OwnCloud * GOA: get OAuth2 tokens out of GNOME Online Accounts "username = goa:..." selects an account in GOA and retrieves the OAuth2 token from that. The implementation uses the GOA D-Bus API directly, because our C++ D-Bus bindings are easier to use and this avoids an additional library dependency. * PIM: fix UID usage in sync.py example Using the underscore in the UID has been wrong all along, it only happened to work because UID sanity checking was missing. After adding it, the example broke. Now simply remove the colon. It makes the UID less readable, but it doesn't have to be, and ensures that file names and database names contain the UID as-is. * PIM: if busy, don't shut down While there are sessions pending or active, the server should not shut down. It did that while executing a long-running PIM Manager SyncPeer() operations, by default after 10 minutes. This was not a problem elsewhere because other operations are associated with a client, whose presence also prevents shutdowns. Perhaps PIM Manager should also track the caller and treat it like a client. * PBAP: do not end Bluez5 transfer prematurely A transfer was marked as finished prematurely when encountering the "active" Status value, which can happen for longer transfers. * updated tests Upgrading from releases <= 1.3.99.4: If the value of "username/databaseUser/proxyUser" contains a colon, the "user:" prefix must be added to the value, to continue treating it like a plain user name and not some reference to an unknown identity provider (like "id:", "goa:", "signon:", etc.). The lookup of passwords in GNOME Keyring was updated slightly in 1.3.99.5. It may be necessary to set passwords anew if the old one is no longer found. Upgrading from release 1.2.x: The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution --configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the "refresh-from-server" sync mode to avoid getting throttled with 417 'retry later' errors. The same must be added to existing configs manually: syncevolution --configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2: Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. SyncEvolution 1.3.99.4, 12.07.2013 ================================== The focus of this development snapshot is enhanced performance of syncing. With EDS, contacts get added, updated or loaded with batch operations, which led to 4x runtime improvements when importing PBAP address book for the first time. Removing unnecessary work from any following PBAP sync led to more than a 6x improvement. These improvements also benefit non-PBAP syncing and could in theory work with any SyncML peer. In practice, batching of items is currently limited to SyncEvolution as peer. The PBAP backend itself was rewritten such that data gets transferred from a phone in parallel to processing the already transferred data. The effect is that on a sufficiently fast system, a sync takes about the same time as downloading all contacts. To get the text-only part of the contacts even faster, PBAP syncing can be done such that it first syncs the text-only parts (without removing existing photos), then in a second round adds or modifies photos. The PIM Manager uses this incremental mode by default, in the command line it can be chose with the SYNCEVOLUTION_PBAP_SYNC env variable. The HTTP server became better at handling message resends when the server is slow with processing a message. The server is able to keep a sync session alive while loading the initial data set by sending acknowledgement replies before the client times out. Guido Günther provided some patches addressing problems when compiling SyncEvolution for Maemo. Details: * sync: less verbose output, shorter runtime For each incoming change, one INFO line with "received x[/out of y]" was printed, immediately followed by another line with total counts "added x, updated y, removed z". For each outgoing change, a "sent x[/out of y]" was printed. In addition, these changes were forwarded to the D-Bus server where a "percent complete" was calculated and broadcasted to clients. All of that caused a very high overhead for every single change, even if the actual logging was off. The syncevo-dbus-server was constantly consuming CPU time during a sync when it should have been mostly idle. To avoid this overhead, the updated received/sent numbers that come from the Synthesis engine are now cached and only processed when done with a SyncML message or some other event happens (whatever happens first). To keep the implementation simple, the "added x, updated y, removed z" information is ignored completely and no longer appears in the output. * HTTP server: handle message resends If a client gave up waiting for the server's response and resent its message while the server was still processing the message, syncing failed with "protocol error: already processing a message" raised by the syncevo-dbus-server because it wasn't prepared to handle that situation. The right place to handle this is inside the syncevo-http-server, because it depends on the protocol (HTTP in this case) whether resending is valid or not. It handles that now by tracking the message that is currently in processing and matching it against each new message. If it matches, the new request replaces the obsolete one without sending the message again to syncevo-dbus-server. When syncevo-dbus-server replies to the old message, the reply is used to finish the newer request. * PBAP: incremental sync (FDO #59551) Depending on the SYNCEVOLUTION_PBAP_SYNC env variable, syncing reads all properties as configured ("all"), excludes photos ("text") or first text, then all ("incremental"). When excluding photos, only known properties get requested. This avoids issues with phones which reject the request when enabling properties via the bit flags. This also helps with "databaseFormat=^PHOTO". * PIM: use incremental sync for PBAP by default (FDO #59551) When doing a PBAP sync, PIM manager asks the D-Bus sync helper to set its SYNCEVOLUTION_PBAP_SYNC to "incremental". If the env variable is already set, it does not get overwritten, which allows overriding this default. * PIM: set debug level in peer configs via env variable Typically the peer configs get created from scratch, in particular when testing with testpim.py. In that case the log level cannot be set in advance and doing it via the D-Bus API is also not supported. Therefore, for debugging, use SYNCEVOLUTION_LOGLEVEL= to create peers with a specific log level. * PIM: include pim-manager-api.txt in source distro (FDO #62516) The text file must be listed explicitly to be included by "make dist". * PIM: "full name" -> "fullname" fix in documentation (FDO #62515) Make the documentation match the code. A single word without space makes more sense, so let's go with what the code already used. * PIM: enhanced searching (search part of FDO #64177) Search terms now also include 'is/contains/begins-with/ends-with' and they can be combined with 'and' and 'or', also recursively. * PIM: Pinyin sorting for zh languages (part of FDO #64173) Full interleaving of Pinyin transliterations of Chinese names with Western names can be done by doing an explicit Pinyin transliteration as part of computing the sort keys. This is done using ICU's Transliteration("Han-Latin"), which we have to call directly because boost::locale does not expose that API. We hard-code this behavior for all "zh" languages (as identified by boost::locale), because by default, ICU would sort Pinyin separately from Western names when using the "pinyin" collation. * PIM: new return value for SyncPeer(), new SyncProgress signal (FDO #63417) The SyncPeer() result is derived from the sync statistics. To have them available, the "sync done" signal must include the SyncReport. Start and end of a sync could already be detected; "modified" signals while a sync runs depends on a new signal inside the SyncContext when switching from one cycle to the next and at the end of the last one. * PIM: allow removal of data together with database removal (part of FDO #64835) There is a difference in EDS between removing the database definition from the ESourceRegistry (which makes the data unaccessible via EDS) and removing the actual database. EDS itself only removes the definition and leaves the data around to be garbage-collected eventually. This is not what we want for the PIM Manager API; the API makes a stronger guarantee that data is really gone. Fixed by introducing a new mode flag for the deleteDatabase() method and deleting the directory of the source directly in the EDS backend, if requested by the caller. The syncevolution command line tool will use the default mode and thus keep the data around, while the PIM Manager forces the removal of data. * EDS: create new databases by cloning the builtin ones (FDO #64176) Instead of hard-coding a specific "Backend Summary Setup" in SyncEvolution, copy the config of the system database. That way special flags (like the desired "Backend Summary Setup" for local address books) can be set on a system-wide basis and without having to modify or configure SyncEvolution. Because EDS has no APIs to clone an ESource or turn a .source file into a new ESource, SyncEvolution has to resort to manipulating and creating the keyfile directly. * EDS contacts: update PHOTO+GEO during slow sync, avoid rewriting PHOTO file If PHOTO and/or GEO were the only modified properties during a slow sync, the updated item was not written into local storage because they were marked as compare="never" = "not relevant". For PHOTO this was intentional in the sample config, with the rationale that local storages often don't store the data exactly as requested. When that happens, comparing the data would lead to unnecessary writes. But EDS and probably all other local SyncEvolution storages (KDE, file) store the photo exactly as requested, so not considering changes had the undesirable effect of not always writing new photo data. For GEO, ignoring it was accidental. * EDS contacts: avoid unnecessary DB writes during slow sync Traditionally, contacts were modified shortly before writing into EDS to match with Evolution expectations (must have N, only one CELL TEL, VOICE flag must be set). During a slow sync, the engine compare the modified contacts with the unmodified, incoming one. This led to mismatches and/or merge operations which end up not changing anything in the DB because the only difference would be removed again before writing. * EDS contacts: read-ahead cache Performance is improved by requesting multiple contacts at once and overlapping reading with processing. On a fast system (SSD, CPU fast enough to not be the limiting factor), testpim.py's testSync takes 8 seconds for a "match" sync where 1000 contacts get loaded and compared against the same set of contacts. Read-ahead with only 1 contact per query speeds that up to 6.7s due to overlapping IO and processing. Read-ahead with the default 50 contacts per query takes 5.5s. It does not get much faster with larger queries. * command line: execute --export and --print-items while the source is still reading Instead of reading all item IDs, then iterating over them, process each new ID as soon as it is available. With sources that support incremental reading (only the PBAP source at the moment) that provides output sooner and is a bit more memory efficient. * WebDAV: avoid segfault during collection lookup Avoid referencing pathProps->second when the set of paths that PROPFINDs returns is empty. Apparently this can happen in combination with Calypso. * engine: prevent timeouts in HTTP server mode HTTP SyncML clients give up after a certain timeout (SyncEvolution after RetryDuration = 5 minutes by default, Nokia e51 after 15 minutes) when the server fails to respond. This can happen with SyncEvolution as server when it uses a slow storage with many items, for example via WebDAV. In the case of slow session startup, multithreading is now used to run the storage initializing in parallel to sending regular "keep-alive" SyncML replies to the client. By default, these replies are sent every 2 minutes. This can be configured with another extensions of the SyncMLVersion property: SyncMLVersion = REQUESTMAXTIME=5m Other modes do not use multithreading by default, but it can be enabled by setting REQUESTMAXTIME explicitly. It can be disabled by setting the time to zero. The new feature depends on a libsynthesis with multithreading enabled and glib >= 2.32.0, which is necessary to make SyncEvolution itself thread-safe. With an older glib, multithreading is disabled, but can be enabled as a stop-gap measure by setting REQUESTMAXTIME explicitly. * Various testing and stability enhancements. SyncEvolution had to be made thread-safe for the HTTP timeout prevention. SyncEvolution 1.3.99.3, 06.03.2013 ================================== Another development snapshot, with a particular focus on enhancing (and in some cases, fixing) searching in the PIM Manager. The PIM Manager in this snapshot depends on folks 0.9.x and thus gee 0.8. Support for Bluez 5 was added. The PIM Manager API was extended by addding CreatePeer and ReplaceSearch. The previous methods, SetPeer and RefineSearch, are still supported. Some issues in CalDAV, WebDAV and SyncML were fixed. Graham R. Cobb contributed several patches for enhancing ActiveSync support and making it work with Exchange 2010. Details: * PIM Manager: add ReplaceSearch, always allow it The new ReplaceSearch is more flexible than RefineSearch. It can handle both tightening the search and relaxing it. The downside of it is the more expensive implementation (must check all contacts again, then find minimal set of change signals to update view). Previously, a search which had no filter set at all at the begining could not be refined. This limitation of the implementation gets removed by always using a FilteredView, even if the initial filter is empty. * PIM Manager: introduce CreateConfig() That SetPeer() allows modifying and creating a config leads to race conditions when multiple clients want to create a config. The new CreateConfig() avoids that by atomically checking that a config does not exist yet and creating it. SetPeer() is still available for backwards compatibility. It continues to be used for modifying an existing config in TestContacts.testSync to check the effect of the logging settings. * PIM Manager: fix double entries in filtered search with limit Stressing the FilteredView by using it in tests originally written for the FullView showed that the filling up a view may have used data while it was inconsistent internally, leading to contacts being present multiple times. * PIM Manager and sync: support location = GEO property (FDO #60373) Exposed as "location" -> (lat, long) in the D-Bus bindings. Reading, writing and updating are supported. * PIM Manager: support groups = CATEGORIES (FDO #60380) Allow reading and writing of groups (folks terminology), aka CATEGORIES in vCard. * PIM Manager: intelligent phone search in EDS (part of FDO #59571) If phone number search is enabled in EDS, then the direct search in EDS now uses the more accurate E_BOOK_QUERY_EQUALS_NATIONAL_PHONE_NUMBER comparison, with the E164 formatted caller ID as value to compare against. This gives semantically correct results. The previous solution (now the fallback) had to use substring searches, which did not match if the contact's phone number was not formatted according to E164 and which may have matched the wrong contacts if the trailing numbers are the same. * PIM Manager : use pre-computed normalized phone numbers from EDS (part of FDO #59571) When available, the pre-computed E164 number from EDS will be used instead of doing one libphonebook parser run for each telephone number while reading. Benchmarking showed that this parsing was the number one hotspot, so this is a considerable improvement. * PIM Manager: fix error messages Ensure and check that no unnecessary ERROR messages are printed. libfolks was used slightly incorrectly, leading to several harmless error messages (glib asserts). libphonenumber printed its error messages to stdout. * PIM Manager: fix memory leaks during writing of contacts Constructing the GValues created additional references instead of taking over ownership as intended. * D-Bus server: fix read-after-free bug when using syslog openlog() expects the string to remain valid. Must ensure that in LoggerSyslog by making a copy. Found with valgrind. * PIM Manager: make implementation of some of the D-Bus methods thread-safe The goal is to make it easier to extend syncevo-dbus-server with other IPC mechanisms, which then can call the native C++ code directly. That code was not prepared to handle calls in threads other than the main one. Now this is checked when entering the methods and work is shifted to the main thread if necessary. In the meantime the calling thread waits for completion. * PIM Manager: check responsiveness (part of FDO #60851) Enhanced the testActive test so that it can detect when the D-Bus server stops responding for too long. One major reason for that was event processing in folks, which got improved as part of https://bugzilla.gnome.org/show_bug.cgi?id=694385 * PIM Manager: adapt to gee 0.8 Changed the code to compile with gee 0.8, as used by folks 0.9.x. Older versions of folks are no longer supported. * PBAP: support Bluez 5 The new Bluez 5 API is the third supported API for doing PBAP transfers. It gets checked first, then the PBAB backend falls back to new-style obexd (file based, similar to Bluez 5, but not quite the same) and finally old-style obexd (data transfer via D-Bus). In contrast to previous APIs, Bluez 5 does not report the reason for a failed PBAP transfer. SyncEvolution then throws a generic "transfer failed" error with "reason unknown" as message. * command line: recover from slow sync with new sync modes The error message for an unexpected slow sync still mentioned the old and obsolete "refresh-from-client/server" sync modes. Better mention "refresh-from-local/remote". * CalDAV: more workarounds for Google CalDAV + unique IDs Google became even more strict about checking REV. Tests which reused a UID after deleting the original item started to fail sometime since middle of December 2012. * CalDAV: work around Google server regression (undeclared namespace prefix in XML) Google CalDAV for a while (December 2012 till January 2013) sent invalid XML back when asked to include CardDAV properties in a PROPFIND. This got rejected in the XML parser, which prevents syncing calendar data: Neon error code 1: XML parse error at line 55: undeclared namespace prefix In the meantime Google fixed the issue in response to a bug report via email. But the workaround, only asking for the properties which are really needed, still makes sense and thus is kept. * WebDAV: don't send Basic Auth via http proactively (FDO #57248) Sending basic authentication headers via http is insecure. Only do it proactively when the connection is encrypted and thus protects the information or when the server explicitly asks for it. * Nokia: always add TYPE=INTERNET to EMAIL (FDO #61784) Without the explicit TYPE=INTERNET, email addresses sent to a Nokia e51 were not shown by the phone and even got lost eventually (when syncing back). This commit ensures that the type is set for all emails sent to any Nokia phone, because there may be other phones which need it and phones which don't, shouldn't mind. This was spot-checked with a N97 mini, which works fine with and without the INTERNET type. This behavior can be disabled again for specific Nokia phones by adding a remote rule which sets the addInternetEmail session variable to FALSE again. Non-Nokia phones can enable the feature in a similar way, by setting the variable to TRUE. * SyncML: config option for broken peers Some peers have problems with meta data (CtCap, old Nokia phones) and the sync mode extensions required for advertising the restart capability (Oracle Beehive). The default in SyncEvolution is to advertise the capability, so manual configuration is necessary when working with a peer that fails in that mode. Because the problem occurs when SyncEvolution contacts the peers before it gets the device information from the peer, dynamic rules based on the peer identifiers cannot be used. Instead the local config must already disable these extra features in advance. The "SyncMLVersion" property gets extended for this. Instead of just "SyncMLVersion = 1.0" (as before) it now becomes possible to say "SyncMLVersion = 1.0, noctcap, norestart". "noctcap" disables sending CtCap. "norestart" disables the sync mode extensions and thus doing multiple sync cycles in the same session (used between SyncEvolution instances in some cases to get client and server into sync in one session). Both keywords are case-insensitive. There's no error checking for typos, so beware! The "SyncMLVersion" property was chosen because it was already in use for configuring SyncML compatibility aspects and adding a new property would have been harder. * ActiveSync: added support for specifying folder names Previously, the database field was interpreted as a Collection ID. This adds logic to allow the database to be interpreted as a folder path. The logic is: 1) If the database is an empty string, pass it through (this is the most common case as it is interpreted as "use the default folder for the source type"). 2) If the database matches a Collection ID, use the ID (this is the same as the previous behaviour). 3) If the database matches a folder path name, with an optional leading "/", use the Collection ID for the matching folder. 4) Otherwise, force a FolderSync to get the latest folder changes from the server and repeat steps 2 and 3 5) If still no match, throw an error. * ActiveSync: support for listing databases Now --print-databases scans folders on the ActiveSync server and shows suitable folders for the ActiveSync backends instead of the previous, hard-coded help text. Invoking --print-databases can be used as a workaround for "SyncFolder error: Invalid synchronization key" errors. A better solution would be to do that automatically, but there was no time to implement that. See FDO #61869 and "[SyncEvolution] Activesync server losing state" http://thread.gmane.org/gmane.comp.mobile.syncevolution/4295 * command line: show backend error when listing databases fails The command line swallowed errors thrown by the backend while listing databases. Instead it just showed ": backend failed". The goal was to not distract users who accidentally access a non-functional backend. But the result is that operations like --configure or --print-databases could fail without giving the user any hint about the root cause of the issue. Now the error explanation in all its gory details is included. For example, not having activesyncd running leads to: INFO] eas_contact: backend failed: fetching folder list: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.meego.activesyncd was not provided by any .service files And running activesyncd without the necessary gconf keys shows up as: [INFO] eas_contact: backend failed: fetching folder list: GDBus.Error:org.meego.activesyncd.Error.AccountNotFound: Failed to find account [syncevolution@lists.intel.com] * Minor memory leak fix when using GDBus GIO: GDBusMethodInfo Also depends on a glib fix, see https://bugzilla.gnome.org/show_bug.cgi?id=695376 * build fixes Avoid -lrt in make dependencies. Add missing pcre libs to syncevo-dbus-server. sqlite backend needs "#include " (patch from Mario Kicherer). SyncEvolution 1.3.99.2, 13.12.2012 ================================== Another development snapshot. Includes all fixes that went into 1.3.2 and several improvements to the PIM Manager. Documentation was updated and extended considerably. The pim-manager-api.txt now describes the abstract API while src/dbus/server/pim/README explains how SyncEvolution implements it. Details: * PIM Manager searches for a caller ID ('phone' search) in EDS directly while folks still starts up. No unification is done of these results. Intermediate results are replaced by the final ones from folks once those are ready. * PIM Manager: allow configuration of session directories (part of FDO #55921) Useful for moving the session directories to a temporary file system. They are essentially just useful for debugging when used as part of PIM Manager. - "logdir" - a directory in which directories are created with debug information about sync session - "maxsessions" - number of sessions that are allowed to exist after a sync (>= 0): 0 is special and means unlimited, 1 for just the latest, etc.; old sessions are pruned heuristically (for example, keep sessions where something changed instead of some where nothing changed), so there is no hard guarantee that the last n sessions are present. * PIM Manager: write less data to disk (part of FDO #55921) Avoid writing config file changes to disk by enabling a new "ephemeral" mode for syncing via the PIM Manager. In this mode, config file changes are not flushed resp. discarded directly. This prevents writing to .ini files in ~/.config. The "synthesis" binfile client files are still written, but they get redirected into the session directory, which can (and should) be set to a temp file system and get deleted again quickly. Data dumps are turned off now in the configs created by the PIM Manager. * syncevo-dbus-server: use syslog instead of standard output by default * syncevo-dbus-server: command line options for controlling output and startup -d, --duration=seconds/'unlimited' Shut down automatically when idle for this duration (default 300 seconds) -v, --verbosity=level Choose amount of output, 0 = no output, 1 = errors, 2 = info, 3 = debug; default is 1. -o, --stdout Enable printing to stdout (result of operations) and stderr (errors/info/debug). -s, --no-syslog Disable printing to syslog. -p, --start-pim Activate the PIM Manager (= unified address book) immediately. * PIM Manager: store set of active address books persistently (FDO #56334) Together with storing the sort order persistently, this allows restarting the daemon and have it create the same unified address book again. * PIM Manager: remove colon from valid peer UID character set (FDO #56436) Using the UID as part of file names gets more problematic when allowing colons. Remove that character from the API and enforce the format in the source code. * PIM Manager API: introduce contact ID and use it for reading This makes it easier for a client to fully polulate its view with contact data. Previously it could happen that due to concurrent changes in the server, a client was returned data for the same contact multiple times. A client had to detect that and re-issue read requests. * PIM Manager API: optional ViewAgent.Quiescent() (FDO #56428) The callback is guaranteed to be invoked once when a search has finished sending its initial results, and not sooner. This makes it possible to check whether the current data contains some contact or not. * PIM Manager: limit number of search results (FDO #56142) A 'limit' search term with a number as parameter (formatted as string) can be added to a 'phone' or 'any-contains' search term to truncate the search results after a certain number of contacts. Example: Search([['any-contains', 'Joe'], ['limit', '10']]) => return the first 10 Joes. As with any other search, the resulting view will be updated if contact data changes. The limit must not be changed in a RefineSearch(). A 'limit' term may (but doesn't have to) be given. If it is given, its value must match the value set when creating the search. This limitation simplifies the implementation and its testing. The limitation could be removed if there is sufficient demand. * PIM Manager: fix refining a search Due to not mapping the local index in the view to the parent's index, refining only worked in views where parent and child had the same index for the contacts in the search view. * PIM Manager: fix starting when done via search When the unified address book (= FullView) was not running yet at the time when a client wanted to search it, the unified address book was not started and thus the search never returned results. * PIM Manager: fix writing contact, support photo and notes folks and EDS do not support writing properties in parallel (https://bugzilla.gnome.org/show_bug.cgi?id=652659). Must serialize setting of modified properties. * PIM Manager: fix incorrect contact removal signals in filtered view The filtered view did not check whether a parent's removed contact was really part of the view before sending a removal signal for it. * D-Bus: missing out parameters in D-Bus introspection XML (FDO #57292) The problem was in the C++ D-Bus binding. If the method that gets bound to D-Bus returns a value, that value was ignored in the signature: int foo() => no out parameter It works when the method was declared as having a retval: void foo (int &result) => integer out parameter This problem existed for both the libdbus and the GIO D-Bus bindings. In SyncEvolution it affected methods like GetVersions(). * PIM Manager performance: pre-compute normalized telephone numbers Looking up by phone number spends most of its cycles in normalizing of the phone numbers in the unified address book. Instead of doing that work over and over again during the search, do it once while loading. Looking up a phone number only once does not gain from this change, it even gets slower (more memory intensive, less cache locality). Only searching multiple times becomes faster. Ultimately it would be best to store the normalized strings together with the telephone number inside EDS when the contact gets created. Work on that is in progress. * PIM Manager: improve performance of FullView sorting This fixes the hotspot during populating the FullView content: moving contacts around required copying IndividualData and thus copying complex C++ structs and strings. Storing pointers and moving those avoids that, with no lack of convenience thanks to boost::ptr_vector. Reordering also becomes faster, because the intermediate copy only needs to be of the pointers instead of the full content. * PIM Manager example: add benchmarking The new "checkpoints" split up the whole script run into pieces which are timed separately, with duration printed to stdout. In addition, tools like "perf" can be started for the duration of one phase. * SyncML: workarounds for broken peers Some peers have problems with meta data (CtCap, old Nokia phones) and the sync mode extensions required for advertising the restart capability (Oracle Beehive). Because the problem occurs when SyncEvolution contacts the peers before it gets the device information from the peer, dynamic rules based on the peer identifiers cannot be used. Instead the local config must already disable these extra features in advance. The "SyncMLVersion" property gets extended for this. Instead of just "SyncMLVersion = 1.0" (as before) it now becomes possible to say "SyncMLVersion = 1.0, noctcap, norestart". "noctcap" disables sending CtCap. "norestart" disables the sync mode extensions and thus doing multiple sync cycles in the same session (used between SyncEvolution instances in some cases to get client and server into sync in one session). Both keywords are case-insensitive. There's no error checking for typos, so beware! The "SyncMLVersion" property was chosen because it was already in use for configuring SyncML compatibility aspects and adding a new property would have been harder. * EDS: fix creating databases --create-database was broken in combination with the final code in EDS 3.6 because it passed NULL for the UID to e_source_new_with_uid(), which is considered an error by the implementation of that method. Must use e_source_new() if we don't have a UID. * fixed some memory leaks, extended tests to cover new features and bugs SyncEvolution 1.3.99.1, 25.10.2012 ================================== Development snapshot. The PIM Manager API implementation is fully implemented, see src/dbus/server/pim/README for an introduction. The PBAP backend together with a new one-way caching sync mode provides an efficient way of keeping a local database in sync via Bluetooth with a phone which does not implement SyncML. Other changes: * workarounds for warnings from g++ 4.5 * engine: : local cache sync mode This patch introduces support for true one-way syncing ("caching"): the local datastore is meant to be an exact copy of the data on the remote side. The assumption is that no modifications are ever made locally outside of syncing. This is different from one-way sync modes, which allows local changes and only temporarily disables sending them to the remote side. Another goal of the new mode is to avoid data writes as much as possible. This new mode only works on the server side of a sync, where the engine has enough control over the data flow. Setting "sync" to: - "local-cache-incremental" will do an incremental sync (if possible) or a slow sync (otherwise). This is usually the right mode to use, and thus has "local-cache" as alias. - "local-cache-slow" will always do a slow sync. Useful for debugging or after (accidentally) making changes on the local side. An incremental sync will ignore such changes because they are not meant to happen, aren't checked for to improve performance and thus will leave client and server out-of-sync! Both modes are recorded in the sync report of the local side. The target side is the client and records the normal "two-way" or "slow" sync modes. With the current SyncEvolution contact field list, first, middle and last name are used to find matches for contacts. For events, tasks and memos, time, summary and description are used. * HTTP proxy: useProxy=0 overrides http_* env variables Previously, if http_proxy was set, a proxy was used even if explicitly disabled. This prevented disabling the use of a proxy which only made sense in some cases, like accessing something that runs locally. Explicitly telling SyncEvolution to ignore http_proxy is necessary because it doesn't support no_proxy. * WebDAV: auto-discovery fix With Google Contact + CardDAV the auto-discovery failed after finding the default address book, without reporting that result. * command line: implement --create/remove-database Creating a database is only possible with a chosen name. The UID is chosen automatically by the storage. Only implemented in the EDS backend. * file backend: sub-second mod time stamps Change tracking in the file backend used to be based on the modification time in seconds. When running many syncs quickly (as in testing), that can lead to changes not being detected when they happen within a second. Now the file backend also includes the sub-second part of the modification time stamp, if available. This change is relevant when upgrading SyncEvolution: most of the items will be considered "updated" once during the first sync after the upgrade (or a downgrade) because the revision strings get calculated differently. * D-Bus server: avoid progress outside of 0-100% range For example in the new TestLocalCache.testItemDelete100, the percentage value in the ProgressChanged signal become larger than 100 and then revert to 100 at the end of the sync. Seems the underlying calculation is faulty or simply inaccurate. This is not fixed. Instead the result is just clipped to the valid range. * code cleanup + improvements in testing SyncEvolution 1.3.1 -> 1.3.2, 05.11.2012 ======================================== Minor (or major, if you depend on auto syncing) bug fix release. Details: * auto sync: only synced once (FDO #56667) A successful sync was incorrectly treated like a sync with a permanent failure, which prevents further automatic syncing. * auto sync: notifications were not translated The code which enabled localization of messages created by the D-Bus server was incomplete. Localization was only enabled accidentally through KDE if the KDE platform modules was enabled during compilation and installed. * HTTP Proxy: useProxy=0 overrides http_* env variables Previously, if http_proxy was set, a proxy was used even if explicitly disabled. This prevented disabling the use of a proxy which only made sense in some cases, like accessing something that runs locally. Explicitly telling SyncEvolution to ignore http_proxy is necessary because it doesn't support no_proxy. * minor changes in testing and autotools files (missing Boost search path in gdbus* libs might have caused compile problems) SyncEvolution 1.3 -> 1.3.1, 05.10.2012 ====================================== Minor bug fix release. Details: * command line: fix output of --import for directories The running count at the start of the line (#0, #1, ...) was not incremented when reading individual files from a directory. * Funambol: work around PHOTO TYPE=image/jpeg, part II The final version of the fix hadn't made it into the source code. * vCalendar 1.0 + tasks: DUE date could be shifted by a day (FDO #55238) Because of incomplete support for time conversion, the due date could get mixed up when phone and PC were set to something other than UTC. Reported and fixed by Peter Jan. * syncevolution.org: syncevolution-evolution had incorrect dependencies Installation on older Linux distros was not possible because the ebook/ecal package dependencies were named incorrectly, for example libebook-1.2-10 instead of libebook1.2-10. Only more recent packages have the extra dash, for example libebook-1.2-12. Reported by Mariusz Sokolowski. * GTK-3 UI: fixed compile problem The GTK-3 UI depends on a class from gio-unix-2.0 and failed to compile on Fedora Core 16 because the configure checks for that lib (and thus the compiler flags) were missing. Reported by Peter Robinson. * Curl: allow using it in the D-Bus server In the past, using curl as HTTP transport in the syncevo-dbus-server was prevented, leading to "unsupported transport type is specified in the configuration". The reason was that using curl would block the server and make it unresponsive on D-Bus. This reason has gone away, because now the HTTP traffic happens in a separate process. Thus now it is allowed to use curl in the syncevo-dbus-server. * fix for false negative in syncevo-dbus-server testing SyncEvolution 1.2.2 -> 1.3, 10.09.2012 ====================================== After almost three months of public beta testing the next major version of SyncEvolution is ready for release. The pre-releases did have the desired effect of flushing out bugs not found by nightly testing alone. Thanks everyone for packaging, downloading and testing them! New features are KDE/Akonadi and ActiveSync support, not only in the source code but also in syncevolution.org binaries. ActiveSync is the recommended way of synchronizing contacts with Google: https://syncevolution.org/wiki/google-contacts-activesync The D-Bus server and local sync were rewritten considerably, to make the code cleaner and more robust. The CalDAV backend now also supports tasks and memos. CalDAV and CardDAV can be used in combination with a SyncML peer ("bridging SyncML and WebDAV"), thus allowing a device which only supports SyncML to talk to a WebDAV service without any intermediate storage. 1.3 contains bug fixes that were not backported to 1.2.x, so upgrading is recommended. For example, SyncEvolution 1.3 is required for Evolution 3.4, otherwise photos are not exported properly. Support for Evolution >= 3.6 is in the source code, but not in syncevolution.org binaries. Further workarounds for recent changes in Google CalDAV and Funambol One Media were added. Details: * ActiveSync: updated to work with latest activesyncd and Google, package binaries Syncing Google contacts was added to the nightly testing. Syncing contacts and events with Exchange 2012 was already working. Setup instructions and known issues are described here: https://syncevolution.org/wiki/google-contacts-activesync * phone sync: delete<->delete conflict + phone calendar+todo sync (BMC #23744) When deleting an item on phone and locally, the next sync failed with ERROR messages about "object not found". This has several reasons: - libsynthesis super data store attempts to read items which may or may not exist (triggers ERROR message) - it checks for 404 but Evolution backends only return a generic database error (causes sync to fail) * phone sync: get phone vendor and model from Device ID profile (BMC #736) In the past we have relied on the user-modifiable device name to be the fingerprint for matching a phone to a template which is unreliable. This release changes this in the cases where the phone supports the Device ID profile (DIP). If support for DIP is detected, then we extract the vendor and product ids and attempt to associate them with a product and vendor name by using a newly added lookup table. This lookup table has to be maintained manually and depends on contributions by users to cover more devices. See http://blixtra.org/blog/2011/09/22/syncevolution-needs-you-or-at-least-your-bluetooth-phones/ * vCalendar 1.0: fixed recurring all-day event support vCalendar 1.0 cannot represent all-day events. The workarounds for mapping iCalendar 2.0 all-day events into vCalendar 1.0 was incomplete, leading to effects like shifting EXDATEs and end times. * Funambol: ignore UID Funambol's OneMedia sends UID, but not RECURRENCE-ID. That becomes a problem when multiple events of the same event series are added to a backend which follows the iCalendar 2.0 standard (CalDAV, EDS, KDE), because these events all look like the master event, and there can be only one of those. SyncEvolution now strips the UID from all events coming from any Funambol server (regardless of the version). If a future Funambol server release adds support for both UID and RECURRENCE-ID, then SyncEvolution will have to be updated to take advantage of the improved server. Because the RECURRENCE-ID is also getting stripped (despite not being set at the moment), SyncEvolution should continue to work as it does now even if the server changes. It would have been nice to limit this workaround to affected Funambol server versions, but an inquiry on the Funambol mailing list didn't get a reply, therefore SyncEvolution is playing it safe and assumes that all future Funambol releases will have the same problem. * Funambol: work around PHOTO TYPE=image/jpeg A combination of Funambol Android and Funambol server recently led to the Funambol server sending PHOTO data with TYPE=image/jpeg. This is invalid and caused EDS to reject the photo (Vladimir Elisseev, "[SyncEvolution] issues with syncing photos"). Work around the problem by only keeping the part of the type after the last slash, if there is any. For image/jpeg and similar types that leads to the desired value and does not affect valid values, because those do not contain a slash (http://www.iana.org/assignments/media-types/image/index.html). * Funambol: avoid slow syncs in refresh from server libsynthesis has traditionally implemented "refresh-from-server" as "delete local data" plus "slow" sync. This is more compatible, because some servers (like Google) do not support "refresh-from-server". But it has the downside that the server cannot know that the client won't send any data, and Funambol's OneMedia now only allows one slow sync before blocking the next one for a certain period of time. This is done to prevent excessive resource usage by badly behaving clients. To accomodate both kinds of servers, the new "enableRefreshSync" sync property can be set set to explicitly allow the usage of the "refresh-from-server" sync mode. It's off by default. The Funambol template has it turned on, existing configs must be updated manually (see upgrading comments below). * Mobical (aka Everdroid): stopped testing memo syncing Memos used to work, but now only trigger an unspecific 400 error on the server side. * GTK-UI: accept service config with a username again (BMC#23106) Suppressing configs with empty username had undesired side effects: modifying configs for direct syncing with a device incorrectly triggered the same error message, without any means of entering a username. The faulty check was removed without replacement. * GTK-UI: added GTK 3 version of UI When GTK 3 is found during compilation, a GTK 3 version of the UI is built. The source code of both is different to avoid excessive use of ifdefs. At the moment, both versions offer the same features. In the long run, the GTK 3 version will replace the GTK 2 version. * command line: added refresh/one-way-from-local/remote (BMC #23537) The -from-client/server sync modes are confusing because the direction of the data exchange depends on which side acts as SyncML server or client. This release introduces new modes which use -from-local/remote instead. The statistics and messages also use these variants now. The old modes are still understood, but are declared as "not recommended" in the documentation. * command line: config and source names are optional (BMC #23783) The need to add "foo" and "bar" pseudo config and source names to the command line even when all parameters for the operation where explicitly specified on the command line was confusing. Now it is possible to invoke item operations without the config and source name. Names which refer to non-existent configs are still accepted, as in previous releases. Typos are handled better by producing a detailed error report which includes (as applicable): - config doesn't exist - source doesn't exist or not selected - backend property not set Because luids used to be positional arguments after and , a new --luids keyword is necessary to indicate that the ensuing parameters are luids and not and . * command line: introduced --print-databases, supported for CalDAV/CardDAV Listing databases is now a dedicated operation, instead of being done whenever syncevolution was invoked without parameters. Advantages: - can be combined with property assignments for backends which do not work without that additional information, for example CalDAV/CardDAV: syncevolution --print-databases \ backend=[caldav|carddav] \ syncURL=... \ username=... \ password=... - can be done for configured sources * command line: use both stdout and stderr Traditionally, the "syncevolution" command line tool mixed its INFO/ERROR/DEBUG messages into the normal stdout. This has the major drawback that error messages get lost during operations like syncevolution --export - @default addressbook | grep "John Doe" Now anything which is not the expected result of the operation is always sent to stderr. Obviously this includes ERROR messages. INFO and DEBUG are harder to decide. Because they usually convey meta information about the running operation, they are also sent to stderr. The output of running a sync goes to both stdout (summary) and stderr (progress). * command line: allow setting empty properties Due to the way how properties were handled internally, it wasn't possible to explicitly set a property to its default value. Instead the property was unset. For example, explicitly setting database= was not possible. This is necessary for client-test and ActiveSync, because client-test needs to know that the testing is expected to run with the default databases (something which normally is avoided by overwriting empty database properties). Now the "is set" state is tracked explicitly in the config storage and command line property APIs. Unsetting a property via the command line could be implemented with an explicit command line option, but is not supported at the moment. * command line: fixed --export When exporting items into a file, the delimiter between items was missing. * command line + local sync: fixed erroneous "Comparison impossible" output. "Comparison impossible" was incorrectly printed after a successful comparison on the target side of local sync. * local sync: fix timeout with local sync with libdbus When using libdbus instead of GIO D-Bus (as done by syncevolution.org binaries and SyncEvolution on Maemo), local sync may have aborted after 25 seconds when syncing many items with a D-Bus timeout error: [ERROR] sending message to child failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible ca Reported by Toke Høiland-Jørgensen for Harmattan. Somehow not encountered elsewhere. * synccompare: shorter data dump of PHOTO A full comparison of the base64 PHOTO data can be very long. Now some key characteristics of the PHOTO data (number of characters in base64 encoding, number of bytes in decoded data, md5sum of decoded data) are printed instead. That way, unintended changes of the data (different encoding, different content) should still be found while testing and added/removed photos are nicely visible in synccompare diffs. * synccompare: fixed output for byte-identical duplicates If database dumps contained byte-identical duplicates, they were treated as a single item on the left side of a comparison. This caused erroneous "added" entries on the right side. * secure password storage: usage of GNOME Keyring vs. KDE KWallet configurable Automatically detecting KDE users is not possible at the moment. Instead KDE users have to manually set the new "keyring" global config property to "KDE" (case insensitive) if the SyncEvolution installation supports both, because GNOME Keyring is the default to avoid surprises for traditional users. If only KWallet support is enabled, then this is not necessary. "GNOME" and "true/false/1/0/yes/no" can also be set. This has the advantage that keyring usage can be enabled permanently for the command line in --daemon=no mode; normally keyrings are not used in that mode because accessing them can bring up UI dialogs. It also becomes possible to disable keyring usage in syncevo-dbus-server, something which couldn't be done before. The --keyring command line option is still supported, as an alias for "[--sync-property] keyring=". The default value for --keyring is true, to match the traditional behavior. In contrast to other sync properties, setting "keyring" does not require an explicit --run parameter. Again this is done to mirror traditional usage. * config: improved 'maxlogdirs' documentation The old explanation made it sound like nothing would get deleted by default ("If set, ..."). That's not correct, by default only 10 sessions are kept. Also explain the behavior of deleting intermediate sessions first. * Evolution: always create databases (PTCOM-113) Always try to create address book or calendar database, because even if there is a source there's no guarantee that the actual database was created already; the original logic for only setting this when explicitly requesting a new database therefore failed in some cases. This problem affected users who had never created anything locally and wanted to use SyncEvolution to migrate their data. Now that works without having to create dummy entries first. * Evolution contacts: changed default sync format to vCard 3.0 vCard 3.0 is the better default because it has saner encoding rules and defines more properties, thus avoiding the need for non-standard extensions. However, Mobical has problems with the new default. See upgrade instructions below. * Evolution: added support for EDS 3.5.x When compiled against EDS 3.5.x or later, SyncEvolution now uses the backend code originally written for the EClient API introduced in EDS 3.2. That code was changed so that it works with the new include file rules and ESourceRegistry in EDS 3.5.x. Support for using the EClient API with EDS 3.4 was removed because maintaining three different flavors of the EDS backend code would be too much work and not gain much (just the possibility to test the EDSClient code with 3.4). At the moment, this is a compile time choice made automatically by configure. syncevolution.org binaries are compiled against an older EDS and thus do not work with EDS 3.5.x or later. EDS 3.5.x handles authentication itself, using a standard system prompt if necessary. SyncEvolution can no longer provide the password, and thus the "databaseUser/Password" options have no effect when using EDS 3.5.x. * D-Bus server: fixed HTTP presence for recent libdbus Testing with libdbus 1.6.0 on Debian Testing failed because the lib changed some behavior: instead of looking up the owner of a certain bus name immediately, it now does that when invoking a method. Therefore the check for "have connection" in SyncEvolution was too simplistic and missed the fact that both were not usable, causing the server to assume that HTTP was down while in reality it should have assumed it to be up. This prevented auto-syncing and manually clicking "Sync" in the GTK UI. * D-Bus server: made notification verbosity configurable with "notifyLevel" The new "notifyLevel" per-peer configuration option allows users to control how many desktop notifications the D-Bus server produces while executing an automatic sync: 0 - suppress all notifications 1 - show only errors 2 - show information about changes and errors (in practice currently the same as level 3) 3 - show all notifications, including starting a sync (default) * WebDAV: fixed data corruption issue when uploading item with long UID In some cases data with a very long UID wasn't handled correctly, causing the out-going data to be malformed and probably causing a rejection by the server. The root cause is incorrect string handling. In releases before 1.2.99.1, only the --import operation of contacts into CardDAV were affected. In 1.2.99.1, the same code also got used for calendar items and then could also affect syncing. * CalDAV: updated Google workarounds Google started sending empty items (VCALENDAR with no VEVENT inside) which cannot be removed. SyncEvolution 1.3 ignores such items. The workaround for a 404 from Google Calendar for a GET (sending a REPORT request matching the item's UID) was broken: first, processing the result ended up calling the unset responseEnd boost function pointer, which caused the request to fail. Second, getting multiple items wasn't handled (data from all items concatenated together was used). That can happen in the somewhat unlike case that some items have a UID which is a complete superset of the requested UID - not realistic in real life, but happens during testing. * Google Calendar: updated URL redirect handling Google Calendar sometimes returns redirect requests to human-readable web sites (an "unavailable" page, a login form). This is of course bogus when the client is an automated CalDAV client. The "unavailable.html" case was already handled. Made it a bit more flexible to also catch possible variations of it (additional parameters, https instead of http). Added the https://accounts.google.com/ServiceLogin case. Not sure whether retrying will help in that case, but there's not much else that SyncEvolution can do. * WebDAV: bridge with SyncML Now a peer accessed via SyncML can read/write data stored in a CalDAV/CardDAV server directly. This can be used to connect a device which only supports SyncML to a CalDAV/CardDAV server, or sync data between a SyncML server and a CalDAV/CardDAV server. See "CalDAV and CardDAV" in the README for details. * WebDAV: improved --configure Added INFO output about checking sources. This helps with WebDAV when the server cannot be contacted (dead, misconfigured) because otherwise there would be no indication at all why the --configure operation seems to hang. Here is some example output, including aborting: $ syncevolution --configure --template webdav \ syncURL=http://192.168.1.100:9000/ \ username=foo password=bar retryDuration=2s \ target-config@webdav-temp [INFO] creating configuration target-config@webdav-temp [INFO] addressbook: looking for databases... [INFO] addressbook: no database to synchronize [INFO] calendar: looking for databases... [INFO] calendar: no database to synchronize [INFO] memo: looking for databases... [INFO] memo: no database to synchronize [INFO] todo: looking for databases... [INFO] todo: no database to synchronize It timed out fairly quickly here because of the retryDuration=2s. That also gets placed in the resulting config, which is probably not desired. Aborting the operation is now supported: $ syncevolution --configure \ --template webdav \ syncURL=http://192.168.1.100:9000/ \ username=foo password=bar \ target-config@webdav-temp [INFO] creating configuration target-config@webdav-temp [INFO] addressbook: looking for databases... ^C[INFO] Asking to suspend... [INFO] Press CTRL-C again quickly (within 2s) to stop immediately (can cause problems in the future!) ^C[INFO] Aborting immediately ... [ERROR] error code from SyncEvolution aborted on behalf of user (local, status 20017): aborting as requested by user It would be good to make the CTRL-C handling code aware that it can abort immediately instead of doing the intermediate "asking to suspend" step, which only makes sense for sync sessions. * WebDAV: support tasks and memos (BMC #24893) The new backend property values "CalDAVTodo" and "CalDAVJournal" select tasks resp. memos stored in a CalDAV collection. "CalDAV" continues to select events. Events, tasks and journals can be mixed in the same resource (= URL). However, this is less efficient than storing them separately. A good CalDAV server allows filtering items by type, and SyncEvolution uses that. However, it was found that Radicale 0.7 ignores this filtering, which could have led to data loss (SyncEvolution asks for all VTODOs in preparation for a "delete all items" operation in a "CalDAVTodo" source, gets also VJOURNALs, then deletes them). Therefore SyncEvolution plays it safe and downloads the VTODO and VJOURNAL data to double-check that it is working on the right items. This causes additional traffic for well-behaving servers; currently it cannot be turned off. Tasks are exchanged as vCalendar 1.0 or iCalendar 2.0 VJOURNAL. Memos are exchanged as VTODO or plain text. The logic for storing incoming plain text is slightly different compared to the way how the EDS memo backend did it: instead of copying the first line from the text into the summary, it is now moved. In other words, the first line gets stripped. The change is primarily technically motivated; both approaches have pros and cons. * WebDAV: improved Radicale support Radicale > 0.7 will return status 200 for delete requests; is now treated like 204 by SyncEvolution. 412 'Preconditiona Failed' when asking to delete an already removed item is treated like the more common 404 'not found'. Same with 410 'gone' instead of 404 when trying to read a non-existent item. * CalDAV/CardDAV sync: improved target side output Added a "target side of local sync ready" INFO message to introduce the output which has the target context in the [INFO] tag. The sync report from the target side now has the target context embedded in brackets after the "Changes applied during synchronization" header, to avoid ambiguities. Sometimes the backend has to resend requests because of temporary issues. If the problem turned out to be permanent, there was a long period of time, retryDuration=5 minutes to be precice, in which no visible progress happened. Now SyncEvolution's WebDAV backend will print a message like this before going to sleep until it is time to retry: [INFO @googlecalendar] operation temporarily (?) failed, going to retry in 5.0s before giving up in 18.4s: PROPFIND: Neon error code 1: 401 Unauthorized The uncertainty comes from several factors. In this example, the 401 might indicate a permanent problem (wrong credentials), or it could be Google reporting a temporary authorization problem which is (probably) meant to slow down the client while it asks the user to re-enter the password. SyncEvolution only asks for passwords once, so it tries again with the same password if it was successful with it in the past. Otherwise it gives up immediately. Another dubious example are name server lookup errors. They can be permanent (wrong host name) or temporary (name server down). SyncEvolution errs on the side of retrying, to avoid interrupting an operation which still has a chance to continue. Output from the target side of a local sync was passed through stderr redirection as chunks of text to the frontends. This had several drawbacks: - forwarding only happened when the local sync parent was processing the output redirection, which (due to limitations of the implementation) only happens when it needs to print something itself - debug messages were not forwarded - message boundaries might have been lost In particular the new INFO messages are relevant while the sync runs and need to be shown immediately. * WebDAV: --status for WebDAV source aborted The command line --status operation did not complete when applied to a CalDAV/CardDAV source. Instead it aborted because the operation took a code path where the backend was not fully initialized. * file backend: more flexible sync support for memos The databaseFormat=text/calendar for memos did not support synchronizing as plain text. When using the new databaseFormat=text/calendar+plain, vCalendar/iCalendar/plain text are all valid sync formats; the storage is iCalendar 2.0 VJOURNAL in all cases. * WebDAV: avoid potential crash during database detection When a server responds to a PROPFIND for a path with results for some other path, then SyncEvolution crashed during the search for the default calendar or address book because of a bug in the code which was meant to handle that kind of response. Apparently Yahoo Calendar did that. Now seen again in combination with Radicale 0.6.4. In general, the code was made more robust to cope with bugs in Radicale 0.6.4. Later Radicale versions fixed these issues and also worked with SyncEvolution 1.2.2 without client-side workarounds. * WebDAV: better path normalization "syncURL" and "database" properties had to end in a trailing slash, otherwise items were not found (404 errors). Now the necessary slash is added automatically. * Curl transport: support SSLServerCertificates= When the setting refers to a directory, then CURLOPT_CAINFO doesn't work (must be a file). Check this and use CURLOPT_CAPATH instead. Caveat: there are some comments in the API documentation about "NSS enabled libcurl" which supports a directory in CURLOPT_CAINFO. Hopefully providing an explicit path in CURLOPT_CAPATH also works in that configuration. * code cleanup + rewrite: syncing done in separate process syncevo-dbus-server now runs syncing in a separate process. Local sync also uses a second helper process. This makes the D-Bus server more responsive via D-Bus (no more blocking operations) and minimizes the effect of bugs in code involved with syncing (backends, system libraries, etc.). In the long term this restructuring will also allow more advanced features, like monitoring local or remote storage for changes. * SyncEvolution <-> SyncEvolution sync: multiple cycles per session SyncML only allows one send/receive cycle per session. There are cases (for example, client side merges data that a dumber server failed to match correctly) where client and server are still out of sync at the end of a cycle. When SyncEvolution syncs with another SyncEvolution instance (locally or remotely), both sides detect that the peer can continue syncing in the same session and start over automatically when needed. Previously the user had to start another sync session manually. To the user this is shown as "number of cycles" in a sync session in the sync report. "Restart" is the process of entering a new cycle. The cycles are also visible in the command line output as multiple INFO lines: [INFO] eds_contact: starting first time sync from client (peer is server) [INFO] creating complete data backup of source eds_contact before sync (enabled with dumpData and needed for prin Local data changes to be applied during synchronization: *** eds_contact *** no changes [INFO] eds_contact: sent 1/1 [INFO] eds_contact: started [INFO] eds_contact: first time sync done successfully [INFO] eds_contact: starting normal sync from client (peer is server) <=== [INFO] eds_contact: started <=== [INFO] eds_contact: normal sync done successfully <=== [INFO] creating complete data backup after sync (enabled with dumpData and needed for printChanges) Synchronization successful. Changes applied during synchronization: +---------------|-----------------------|-----------------------|-CON-+ | | LOCAL | REMOTE | FLI | | Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS | +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ | eds_contact | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | | refresh-from-local, 2 cycles, 0 KB sent by client, 0 KB received | ^^^^^^^^ | item(s) in database backup: 1 before sync, 1 after it | +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ | start Tue Feb 7 17:07:49 2012, duration 0:03min | | synchronization completed successfully | +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ * SyncEvolution <-> SyncEvolution sync: negotiate UID support via SyncCap (BMC #22783) The semantic of UID/RECURRENCE-ID in calendar data is now tracked per data store involved in a sync. If full iCalendar 2.0 semantic (= IDs are globally unique) is guaranteed, then pairs are found based on these IDs. Otherwise pairs must be found by looking at item attributes. Previously a hack was used to detect this kind of support (any kind of SyncEvolution instance was assumed to support it, although some backends do not). * engine: add DTSTAMP+LAST-MODIFIED before writing calendar items When writing calendar items into a backend storage as iCalendar 2.0 or vCalendar 1.0, they should have DTSTAMP and LAST-MODIFIED values. DTSTAMP is expected by some CalDAV servers (like Apple). LAST-MODIFIED is usually added by the storage, but not always. In the text/plain -> syncevolution -> text/calendar -> Radicale -> EDS -> syncevolution chain the LAST-MODIFIED was not added by Radicale, which caused problems for change tracking in an EDS-based SyncEvolution. Also necessary when importing from a phone using vCalendar without DTSTAMP directly into CalDAV. * autotools: ensure that link lines are complete As mentioned by Tino Keitel on the mailing list, some libs and executables were only implicitly linked against libraries that they called directly. This happened to work by chance because these libraries ended up in the running executable anyway, due to indirect loading. Now there is a "make installcheck" test for this kind of defect and the makefiles were updated to avoid it. One exception is libsmltk, which depends on the caller providing SySync logging support. * syncevolution.org packages: fixed D-Bus server autostart in .deb and .rpm packages syncevo-dbus-server wasn't started automatically as part of a user session because /etc/xdg/autostart/syncevo-dbus-server.desktop wasn't included in the packages. This broke auto syncing after a session restart (required manually starting SyncEvolution). * syncevolution.org packages: support KDE The traditional "syncevolution-evolution" package was replaced with "syncevolution-bundle". A meta "syncevolution-evolution" package depends on it, to support seamless updates for users who have "syncevolution-evolution" installed. Binary dependencies of the main .deb are ignored for backends because loading them is optional. The new "syncevolution-kde" package has the right dependencies for KDE/Akonadi, while "syncevolution-evolution" mostly just lists standard libs if the "EDS compatibility" mode is used, where libebook/libecal are loaded dynamically. Platform specific code (GNOME keyring, KDE wallet) was moved into loadable, optional modules, to allow installation of the SyncEvolution bundle without forcing the installation of unused system components. * D-Bus: use GIO D-Bus instead of libdbus if available When compiling from source, the more modern GIO D-Bus is used instead of libdbus if available and recent enough (>= 2.30). syncevolution.org binaries still use libdbus, to stay compatible with older Linux distros. * several minor bug fixes syncevo-dbus-server now runs under valgrind in the nightly testing, plus several more test scenarios were added. This helped to find and fix various minor memory handling issues. * developers: backend API changes beginSync/endSync() (aka m_startDataRead/m_endDataWrite) may now be called multiple times per SyncSource instance life cycle. SyncSources derived from TrackingSyncSource should work without changes. Use the Client::Source::*::testChangesMultiCycles test to check whether your backend supports this correctly. Reading and deleting must throw a 404 status exception when an item is not found. The Client::Source::*::*404 tests cover this. The special semantic of the former RegisterSyncSource::InactiveSource (invalid pointer of value 1) caused bugs, like using it in --print-databases (=> segfault) or not being able to store the result of a createSource() directly in a smart pointer (=> potential leak in SyncSource::createSource()). Obviously a bad idea to start with. Replaced with a RegisterSyncSource::InactiveSource() method which creates a real, inactive SyncSource instance which can and must be deleted by the caller. This is a SyncSource API change for backend developers. Instead of RegisterSyncSource::InactiveSource, return RegisterSyncSource::InactiveSource(). Comparisons against RegisterSyncSource::InactiveSource needs to be replaced with a call to the new SyncSource::isInactive(). Long-running backend calls are encouraged to check for events on the main glib context (either in a loop or with g_main_context_iteration(NULL)) and abort when SuspendFlags::getSuspendFlags().getState() returns SuspendFlags::ABORT. Implementing the improved local sync output required extending the D-Bus API. The Server.LogOutput signal now has an additional "process name" parameter. Normally it is empty. For messages originating from the target side, it carries that extra target context string. This D-Bus API change is backward compatible. Older clients can still subscribe to and decode the LogOutput messages, they'll simply ignore the extra parameter. Newer clients expecting that extra parameter won't work with an older D-Bus daemon: they'll fail to decode the D-Bus message. * packagers: libgdbussyncevo is now installed as a normal library in /usr/lib, even though SyncEvolution is the only user. pcrecpp is now a new hard dependency. Upgrading from release 1.2.x: The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution --configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the "refresh-from-server" sync mode to avoid getting throttled with 417 'retry later' errors. The same must be added to existing configs manually: syncevolution --configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2: Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. SyncEvolution 1.2.99.3 -> 1.3, 10.09.2012 ========================================= Final SyncEvolution 1.3 release. The pre-releases did have the desired effect of flushing out bugs not found by nightly testing alone. Thanks everyone for packaging, downloading and testing them! Time to get it out officially as the next stable release. * D-Bus server + GIO D-Bus: shutdown fix When compiled against GIO D-Bus (not the case in syncevolution.org binaries), the syncevo-dbus-server occasionally shut down before sending out all pending D-Bus messages. Showed up only in nightly testing. * D-Bus server + GIO D-Bus: fix auto-activation (Debian bug #599247) When syncevo-dbus-server was started on demand by the D-Bus daemon, then it registered itself with the daemon before it was ready to serve requests. Only happened in combination with GIO D-Bus and thus was not a problem before 1.2.99.x. One user-visible effect was that the GTK UI did not select the default service when it was started for the first time, because it could not retrieve that information from syncevo-dbus-server. * local sync: fix timeout with local sync with libdbus When using libdbus instead of GIO D-Bus (as done by syncevolution.org binaries and SyncEvolution on Maemo), local sync may have aborted after 25 seconds when syncing many items with a D-Bus timeout error: [ERROR] sending message to child failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible ca Reported by Toke Høiland-Jørgensen for Harmattan. Somehow not encountered elsewhere. * KDE: check for D-Bus to avoid crash in KApplication (BMC #25596) Some unnamed version of KDE crashes in KApplication when invoked without a D-Bus session. The reporter ran into this when compiling from source, because the SyncEvolution binary is invoked as part of the build process, which ran outside of a D-Bus session. Avoid the crash by checking for a D-Bus session bus before instantiating KApplication. Instantiating KApplication was added for KWallet support. Without D-Bus, KWallet does not work either, therefore throw an explicit error when the lack of D-Bus is detected. * Funambol: work around PHOTO TYPE=image/jpeg A combination of Funambol Android and Funambol server recently led to the Funambol server sending PHOTO data with TYPE=image/jpeg. This is invalid and caused EDS to reject the photo (Vladimir Elisseev, "[SyncEvolution] issues with syncing photos"). Work around the problem by only keeping the part of the type after the last slash, if there is any. For image/jpeg and similar types that leads to the desired value and does not affect valid values, because those do not contain a slash (http://www.iana.org/assignments/media-types/image/index.html). * syncevo-http-server: fixed printing of server debug output Python failed to call logSyncEvoOutput() after adding the additional 'process' parameter to LogOutput because it extracts all four parameters and then cannot pass them to logSyncEvoOutput(). Now logSyncEvoOutput() uses the new process information to instantiate a logger with the right prefix, using 'sync' as fallback for messages without that information (as before). * Some minor code and test cleanup. SyncEvolution 1.2.99.3 -> 1.2.99.4, 07.08.2012 ============================================== Another release candidate for SyncEvolution 1.3. Lesson learned: declaring a snapshot as "final" is a good way of luring the hidden bugs into the light. Of course, then another snapshot is needed... Details: * D-Bus server: fix support for autoSyncDelay > 0 Auto syncing was not getting triggered when using an autoSyncDelay > 0; by default it is 5 minutes. Thanks to Vladimir Elisseev for reporting this problem. * command line: fixed --export When exporting items into a file, the delimiter between items was missing. * config: improved 'maxlogdirs' documentation The old explanation made it sound like nothing would get deleted by default ("If set, ..."). That's not correct, by default only 10 sessions are kept. Also explain the behavior of deleting intermediate sessions first. * developers: fixed D-Bus interface XML Reverted to Qt 4.x compatible annotations and changed "templateName" to "getTemplate" to make it more obvious what the parameter does. Only relevant for the out-of-tree Qt UI. Fixed accidental removal of the "template" parameter in Session.GetNamedConfig(). Was not used in practice, but has to be correct in case that someone wants to use it. SyncEvolution 1.2.99.2 -> 1.2.99.3, 24.07.2012 ============================================== Final release candidate for SyncEvolution 1.3 - fingers crossed, knock on wood, etc. ActiveSync is now available in binaries from syncevolution.org and becomes the recommended way of synchronizing contacts with Google. EDS 3.5.x and later are supported when compiling from source; syncevolution.org binaries continue to support only EDS up to 3.4. Details: * EDS: added support for EDS 3.5.x When compiled against EDS 3.5.x or later, SyncEvolution now uses the backend code originally written for the EClient API introduced in EDS 3.2. That code was changed so that it works with the new include file rules and ESourceRegistry in EDS 3.5.x. Support for using the EClient API with EDS 3.4 was removed because maintaining three different flavors of the EDS backend code would be too much work and not gain much (just the possibility to test the EDSClient code with 3.4). At the moment, this is a compile time choice made automatically by configure. syncevolution.org binaries are compiled against an older EDS and thus do not work with EDS 3.5.x or later. EDS 3.5.x handles authentication itself, using a standard system prompt if necessary. SyncEvolution can no longer provide the password, and thus the "databaseUser/Password" options have no effect when using EDS 3.5.x. * ActiveSync: updated to work with latest activesyncd and Google, package binaries Syncing Google contacts was added to the nightly testing. Syncing contacts and events with Exchange 2012 was already working. Setup instructions and known issues are described here: https://syncevolution.org/wiki/google-contacts-activesync * local sync: don't drop data comparison output on target side synccompare on the target side of a local sync was invoked with its output being redirected via an unreliable socket to the local sync parent. When the output was large, some of it might have been lost. * local sync: fixed crash When processing stdout from syncevo-local-child in syncevo-dbus-helper, the LogRedirect class was invoked recursively and tried to print the same stdout data repeatedly until the syncevo-dbus-helper crashed due to the infinite recurssion. * local sync: fixed helper process shutdown in case of parent failure The helper process only detected that the parent failed when it tried to log something while the parent had already shut down the D-Bus connection. Even that did not work reliably and differed between D-Bus libdbus and GIO. Added several test cases and fixes for "process died prematurely" error scenarios. * Mobical (aka Everdroid): stopped testing memo syncing Memos used to work, but now only trigger an unspecific 400 error on the server side. * autotools: ensure that link lines are complete As mentioned by Tino Keitel on the mailing list, some libs and executables were only implicitly linked against libraries that they called directly. This happened to work by chance because these libraries ended up in the running executable anyway, due to indirect loading. Now there is a "make installcheck" test for this kind of defect and the makefiles were updated to avoid it. One exception is libsmltk, which depends on the caller providing SySync logging support. * D-Bus server: fixed HTTP presence for recent libdbus Testing with libdbus 1.6.0 on Debian Testing failed because the lib changed some behavior: instead of looking up the owner of a certain bus name immediately, it now does that when invoking a method. Therefore the check for "have connection" in SyncEvolution was too simplistic and missed the fact that both were not usable, causing the server to assume that HTTP was down while in reality it should have assumed it to be up. This prevented auto-syncing and manually clicking "Sync" in the GTK UI. * syncevolution.org: declare dependencies on libical and EDS Let the bundle .deb depend on libical if the lib was enabled during compilation (for example, for CalDAV). This ensures that it gets installed on systems which otherwise don't have it. "syncevolution-evolution" is compatible (and depends on) EDS up to and including 3.4. The package now declares that dependency and conflicts with more recent EDS, because even if the older EDS libs are still installed they won't work when the rest of EDS was updated. * CalDAV + syncevolution.org: fixed segfault without libical+libecal When libical and libecal were not installed, trying to use the CalDAV backend for VEVENTs segfaulted because it depends on libical and did not check properly for it. Only affected syncevolution.org binaries. SyncEvolution 1.2.99.1 -> 1.2.99.2, 04.07.2012 ============================================== Next step towards SyncEvolution 1.3. It adds a workaround for Funambol's OneMedia and fixes an old bug which became more severe in 1.2.99.1. Also has some usability improvements for CalDAV/CardDAV. Hopefully it will not take long to stabilize the code, so test it now while it is still hot :-) Details: * Funambol: ignore UID Funambol's OneMedia sends UID, but not RECURRENCE-ID. That becomes a problem when multiple events of the same event series are added to a backend which follows the iCalendar 2.0 standard (CalDAV, EDS, KDE), because these events all look like the master event, and there can be only one of those. SyncEvolution now strips the UID from all events coming from any Funambol server (regardless of the version). If a future Funambol server release adds support for both UID and RECURRENCE-ID, then SyncEvolution will have to be updated to take advantage of the improved server. Because the RECURRENCE-ID is also getting stripped (despite not being set at the moment), SyncEvolution should continue to work as it does now even if the server changes. It would have been nice to limit this workaround to affected Funambol server versions, but an inquiry on the Funambol mailing list didn't get a reply, therefore SyncEvolution is playing it safe and assumes that all future Funambol releases will have the same problem. * WebDAV: fixed data corruption issue when uploading item with long UID In some cases data with a very long UID wasn't handled correctly, causing the out-going data to be malformed and probably causing a rejection by the server. The root cause is incorrect string handling. In releases before 1.2.99.1, only the --import operation of contacts into CardDAV were affected. In 1.2.99.1, the same code also got used for calendar items and then could also affect syncing. * engine: add DTSTAMP+LAST-MODIFIED before writing calendar items When writing calendar items into a backend storage as iCalendar 2.0 or vCalendar 1.0, they should have DTSTAMP and LAST-MODIFIED values. DTSTAMP is expected by some CalDAV servers (like Apple). LAST-MODIFIED is usually added by the storage, but not always. In the text/plain -> syncevolution -> text/calendar -> Radicale -> EDS -> syncevolution chain the LAST-MODIFIED was not added by Radicale, which caused problems for change tracking in an EDS-based SyncEvolution. Also necessary when importing from a phone using vCalendar without DTSTAMP directly into CalDAV. * Google Calendar: updated URL redirect handling Google Calendar sometimes returns redirect requests to human-readable web sites (an "unavailable" page, a login form). This is of course bogus when the client is an automated CalDAV client. The "unavailable.html" case was already handled. Made it a bit more flexible to also catch possible variations of it (additional parameters, https instead of http). Added the https://accounts.google.com/ServiceLogin case. Not sure whether retrying will help in that case, but there's not much else that SyncEvolution can do. * CalDAV + VJOURNAL: handle UID conflicts When asked to insert a VJOURNAL which already existed (= same UID), CalDAV servers respond with a 412 "Precondition failed" error. This needs to be detected and translated into an "item needs to be merged" result so that the engine can load the existing item, merge the data, and then write back. * WebDAV: --status for WebDAV source aborted The command line --status operation did not complete when applied to a CalDAV/CardDAV source. Instead it aborted because the operation took a code path where the backend was not fully initialized. * CalDAV/CardDAV sync: improved target side output Added a "target side of local sync ready" INFO message to introduce the output which has the target context in the [INFO] tag. The sync report from the target side now has the target context embedded in brackets after the "Changes applied during synchronization" header, to avoid ambiguities. Sometimes the backend has to resend requests because of temporary issues. If the problem turned out to be permanent, there was a long period of time, retryDuration=5 minutes to be precice, in which no visible progress happened. Now SyncEvolution's WebDAV backend will print a message like this before going to sleep until it is time to retry: [INFO @googlecalendar] operation temporarily (?) failed, going to retry in 5.0s before giving up in 18.4s: PROPFIND: Neon error code 1: 401 Unauthorized The uncertainty comes from several factors. In this example, the 401 might indicate a permanent problem (wrong credentials), or it could be Google reporting a temporary authorization problem which is (probably) meant to slow down the client while it asks the user to re-enter the password. SyncEvolution only asks for passwords once, so it tries again with the same password if it was successful with it in the past. Otherwise it gives up immediately. Another dubious example are name server lookup errors. They can be permanent (wrong host name) or temporary (name server down). SyncEvolution errs on the side of retrying, to avoid interrupting an operation which still has a chance to continue. Output from the target side of a local sync was passed through stderr redirection as chunks of text to the frontends. This had several drawbacks: - forwarding only happened when the local sync parent was processing the output redirection, which (due to limitations of the implementation) only happens when it needs to print something itself - debug messages were not forwarded - message boundaries might have been lost In particular the new INFO messages are relevant while the sync runs and need to be shown immediately. * command line: fixed password + property lookup during --print-databases --print-databases for an existing configuration did not look up passwords stored in a keyring, causing the operation to fail for backends like CalDAV/CardDAV where credentials are required. Overriding source properties in that case also only worked when using the unqualified property name ("databasePassword=foo") but not when using the source name as prefix ("calendar/databasePassword=foo"). * Developers: Implementing the improved local sync output required extending the D-Bus API. The Server.LogOutput signal now has an additional "process name" parameter. Normally it is empty. For messages originating from the target side, it carries that extra target context string. This D-Bus API change is backward compatible. Older clients can still subscribe to and decode the LogOutput messages, they'll simply ignore the extra parameter. Newer clients expecting that extra parameter won't work with an older D-Bus daemon: they'll fail to decode the D-Bus message. SyncEvolution 1.2.2 -> 1.2.99.1, 22.06.2012 =========================================== First pre-release of SyncEvolution 1.3. Contains bug fixes that were not backported to 1.2.x, so upgrading is recommended. For example, SyncEvolution 1.3 is required for Evolution 3.4, otherwise photos are not exported properly. Further workarounds for recent changes in Google CalDAV were added. Major new features are KDE/Akonadi support in the syncevolution.org binaries and ActiveSync support (only in the source code). The D-Bus server and local sync were rewritten considerably, to make the code cleaner and more robust. The CalDAV backend now also supports tasks and memos. Details: * phone sync: delete<->delete conflict + phone calendar+todo sync (BMC #23744) When deleting an item on phone and locally, the next sync failed with ERROR messages about "object not found". This has several reasons: - libsynthesis super data store attempts to read items which may or may not exist (triggers ERROR message) - it checks for 404 but Evolution backends only return a generic database error (causes sync to fail) * phone sync: get phone vendor and model from Device ID profile (BMC #736) In the past we have relied on the user-modifiable device name to be the fingerprint for matching a phone to a template which is unreliable. This release changes this in the cases where the phone supports the Device ID profile (DIP). If support for DIP is detected, then we extract the vendor and product ids and attempt to associate them with a product and vendor name by using a newly added lookup table. This lookup table has to be maintained manually and depends on contributions by users to cover more devices. See http://blixtra.org/blog/2011/09/22/syncevolution-needs-you-or-at-least-your-bluetooth-phones/ * vCalendar 1.0: fixed recurring all-day event support vCalendar 1.0 cannot represent all-day events. The workarounds for mapping iCalendar 2.0 all-day events into vCalendar 1.0 was incomplete, leading to effects like shifting EXDATEs and end times. * GTK-UI: accept service config with a username again (BMC#23106) Suppressing configs with empty username had undesired side effects: modifying configs for direct syncing with a device incorrectly triggered the same error message, without any means of entering a username. The faulty check was removed without replacement. * GTK-UI: added GTK 3 version of UI When GTK 3 is found during compilation, a GTK 3 version of the UI is built. The source code of both is different to avoid excessive use of ifdefs. At the moment, both versions offer the same features. In the long run, the GTK 3 version will replace the GTK 2 version. * command line: added refresh/one-way-from-local/remote (BMC #23537) The -from-client/server sync modes are confusing because the direction of the data exchange depends on which side acts as SyncML server or client. This release introduces new modes which use -from-local/remote instead. The statistics and messages also use these variants now. The old modes are still understood, but are declared as "not recommended" in the documentation. * command line: config and source names are optional (BMC #23783) The need to add "foo" and "bar" pseudo config and source names to the command line even when all parameters for the operation where explicitly specified on the command line was confusing. Now it is possible to invoke item operations without the config and source name. Names which refer to non-existent configs are still accepted, as in previous releases. Typos are handled better by producing a detailed error report which includes (as applicable): - config doesn't exist - source doesn't exist or not selected - backend property not set Because luids used to be positional arguments after and , a new --luids keyword is necessary to indicate that the ensuing parameters are luids and not and . * command line: introduced --print-databases, supported for CalDAV/CardDAV Listing databases is now a dedicated operation, instead of being done whenever syncevolution was invoked without parameters. Advantages: - can be combined with property assignments for backends which do not work without that additional information, for example CalDAV/CardDAV: syncevolution --print-databases \ backend=[caldav|carddav] \ syncURL=... \ username=... \ password=... - can be done for configured sources * command line: use both stdout and stderr Traditionally, the "syncevolution" command line tool mixed its INFO/ERROR/DEBUG messages into the normal stdout. This has the major drawback that error messages get lost during operations like syncevolution --export - @default addressbook | grep "John Doe" Now anything which not the expected result of the operation is always sent to stderr. Obviously this includes ERROR messages. INFO and DEBUG are harder to decide. Because they usually convey meta information about the running operation, they are also sent to stderr. The output of running a sync goes to both stdout (summary) and stderr (progress). * command line: allow setting empty properties Due to the way how properties were handled internally, it wasn't possible to explicitly set a property to its default value. Instead the property was unset. For example, explicitly setting database= was not possible. This is necessary for client-test and ActiveSync, because client-test needs to know that the testing is expected to run with the default databases (something which normally is avoided by overwriting empty database properties). Now the "is set" state is tracked explicitly in the config storage and command line property APIs. Unsetting a property via the command line could be implemented with an explicit command line option, but is not supported at the moment. * command line + local sync: fixed erroneous "Comparison impossible" output. "Comparison impossible" was incorrectly printed after a successful comparison on the target side of local sync. * synccompare: shorter data dump of PHOTO A full comparison of the base64 PHOTO data can be very long. Now some key characteristics of the PHOTO data (number of characters in base64 encoding, number of bytes in decoded data, md5sum of decoded data) are printed instead. That way, unintended changes of the data (different encoding, different content) should still be found while testing and added/removed photos are nicely visible in synccompare diffs. * synccompare: fixed output for byte-identical duplicates If database dumps contained byte-identical duplicates, they were treated as a single item on the left side of a comparison. This caused erroneous "added" entries on the right side. * secure password storage: usage of GNOME Keyring vs. KDE KWallet configurable Automatically detecting KDE users is not possible at the moment. Instead KDE users have to manually set the new "keyring" global config property to "KDE" (case insensitive) if the SyncEvolution installation supports both, because GNOME Keyring is the default to avoid surprises for traditional users. If only KWallet support is enabled, then this is not necessary. "GNOME" and "true/false/1/0/yes/no" can also be set. This has the advantage that keyring usage can be enabled permanently for the command line in --daemon=no mode; normally keyrings are not used in that mode because accessing them can bring up UI dialogs. It also becomes possible to disable keyring usage in syncevo-dbus-server, something which couldn't be done before. The --keyring command line option is still supported, as an alias for "[--sync-property] keyring=". The default value for --keyring is true, to match the traditional behavior. In contrast to other sync properties, setting "keyring" does not require an explicit --run parameter. Again this is done to mirror traditional usage. * Evolution: always create databases (PTCOM-113) Always try to create address book or calendar database, because even if there is a source there's no guarantee that the actual database was created already; the original logic for only setting this when explicitly requesting a new database therefore failed in some cases. This problem affected users who had never created anything locally and wanted to use SyncEvolution to migrate their data. Now that works without having to create dummy entries first. * Evolution contacts: changed default sync format to vCard 3.0 vCard 3.0 is the better default because it has saner encoding rules and defines more properties, thus avoiding the need for non-standard extensions. However, Mobical has problems with the new default. See upgrade instructions below. * D-Bus server: made notification verbosity configurable with "notifyLevel" The new "notifyLevel" per-peer configuration option allows users to control how many desktop notifications the D-Bus server produces while executing an automatic sync: 0 - suppress all notifications 1 - show only errors 2 - show information about changes and errors (in practice currently the same as level 3) 3 - show all notifications, including starting a sync (default) * CalDAV: updated Google workarounds Google started sending empty items (VCALENDAR with no VEVENT inside) which cannot be removed. SyncEvolution 1.3 ignores such items. The workaround for a 404 from Google Calendar for a GET (sending a REPORT request matching the item's UID) was broken: first, processing the result ended up calling the unset responseEnd boost function pointer, which caused the request to fail. Second, getting multiple items wasn't handled (data from all items concatenated together was used). That can happen in the somewhat unlike case that some items have a UID which is a complete superset of the requested UID - not realistic in real life, but happens during testing. * WebDAV: bridge with SyncML Now a peer accessed via SyncML can read/write data stored in a CalDAV/CardDAV server directly. This can be used to connect a device which only supports SyncML to a CalDAV/CardDAV server, or sync data between a SyncML server and a CalDAV/CardDAV server. See "CalDAV and CardDAV" in the README for details. * WebDAV: improved --configure Added INFO output about checking sources. This helps with WebDAV when the server cannot be contacted (dead, misconfigured) because otherwise there would be no indication at all why the --configure operation seems to hang. Here is some example output, including aborting: $ syncevolution --configure --template webdav \ syncURL=http://192.168.1.100:9000/ \ username=foo password=bar retryDuration=2s \ target-config@webdav-temp [INFO] creating configuration target-config@webdav-temp [INFO] addressbook: looking for databases... [INFO] addressbook: no database to synchronize [INFO] calendar: looking for databases... [INFO] calendar: no database to synchronize [INFO] memo: looking for databases... [INFO] memo: no database to synchronize [INFO] todo: looking for databases... [INFO] todo: no database to synchronize It timed out fairly quickly here because of the retryDuration=2s. That also gets placed in the resulting config, which is probably not desired. Aborting the operation is now supported: $ syncevolution --configure \ --template webdav \ syncURL=http://192.168.1.100:9000/ \ username=foo password=bar \ target-config@webdav-temp [INFO] creating configuration target-config@webdav-temp [INFO] addressbook: looking for databases... ^C[INFO] Asking to suspend... [INFO] Press CTRL-C again quickly (within 2s) to stop immediately (can cause problems in the future!) ^C[INFO] Aborting immediately ... [ERROR] error code from SyncEvolution aborted on behalf of user (local, status 20017): aborting as requested by user It would be good to make the CTRL-C handling code aware that it can abort immediately instead of doing the intermediate "asking to suspend" step, which only makes sense for sync sessions. * WebDAV: support tasks and memos (BMC #24893) The new backend property values "CalDAVTodo" and "CalDAVJournal" select tasks resp. memos stored in a CalDAV collection. "CalDAV" continues to select events. Events, tasks and journals can be mixed in the same resource (= URL). However, this is less efficient than storing them separately. A good CalDAV server allows filtering items by type, and SyncEvolution uses that. However, it was found that Radicale 0.7 ignores this filtering, which could have led to data loss (SyncEvolution asks for all VTODOs in preparation for a "delete all items" operation in a "CalDAVTodo" source, gets also VJOURNALs, then deletes them). Therefore SyncEvolution plays it safe and downloads the VTODO and VJOURNAL data to double-check that it is working on the right items. This causes additional traffic for well-behaving servers; currently it cannot be turned off. Tasks are exchanged as vCalendar 1.0 or iCalendar 2.0 VJOURNAL. Memos are exchanged as VTODO or plain text. The logic for storing incoming plain text is slightly different compared to the way how the EDS memo backend did it: instead of copying the first line from the text into the summary, it is now moved. In other words, the first line gets stripped. The change is primarily technically motivated; both approaches have pros and cons. * WebDAV: improved Radicale support Radicale > 0.7 will return status 200 for delete requests; is now treated like 204 by SyncEvolution. 412 'Preconditiona Failed' when asking to delete an already removed item is treated like the more common 404 'not found'. Same with 410 'gone' instead of 404 when trying to read a non-existent item. * file backend: more flexible sync support for memos The databaseFormat=text/calendar for memos did not support synchronizing as plain text. When using the new databaseFormat=text/calendar+plain, vCalendar/iCalendar/plain text are all valid sync formats; the storage is iCalendar 2.0 VJOURNAL in all cases. * WebDAV: avoid potential crash during database detection When a server responds to a PROPFIND for a path with results for some other path, then SyncEvolution crashed during the search for the default calendar or address book because of a bug in the code which was meant to handle that kind of response. Apparently Yahoo Calendar did that. Now seen again in combination with Radicale 0.6.4. In general, the code was made more robust to cope with bugs in Radicale 0.6.4. Later Radicale versions fixed these issues and also worked with SyncEvolution 1.2.2 without client-side workarounds. * WebDAV: better path normalization "syncURL" and "database" properties had to end in a trailing slash, otherwise items were not found (404 errors). Now the necessary slash is added automatically. * Funambol: avoid slow syncs in refresh from server libsynthesis has traditionally implemented "refresh-from-server" as "delete local data" plus "slow" sync. This is more compatible, because some servers (like Google) do not support "refresh-from-server". But it has the downside that the server cannot know that the client won't send any data, and Funambol's OneMedia now only allows one slow sync before blocking the next one for a certain period of time. This is done to prevent excessive resource usage by badly behaving clients. To accomodate both kinds of servers, the new "enableRefreshSync" sync property can be set set to explicitly allow the usage of the "refresh-from-server" sync mode. It's off by default. The Funambol template has it turned on, existing configs must be updated manually (see upgrading comments below). * Curl transport: support SSLServerCertificates= When the setting refers to a directory, then CURLOPT_CAINFO doesn't work (must be a file). Check this and use CURLOPT_CAPATH instead. Caveat: there are some comments in the API documentation about "NSS enabled libcurl" which supports a directory in CURLOPT_CAINFO. Hopefully providing an explicit path in CURLOPT_CAPATH also works in that configuration. * code cleanup + rewrite: syncing done in separate process syncevo-dbus-server now runs syncing in a separate process. Local sync also uses a second helper process. This makes the D-Bus server more responsive via D-Bus (no more blocking operations) and minimizes the effect of bugs in code involved with syncing (backends, system libraries, etc.). In the long term this restructuring will also allow more advanced features, like monitoring local or remote storage for changes. * SyncEvolution <-> SyncEvolution sync: multiple cycles per session SyncML only allows one send/receive cycle per session. There are cases (for example, client side merges data that a dumber server failed to match correctly) where client and server are still out of sync at the end of a cycle. When SyncEvolution syncs with another SyncEvolution instance (locally or remotely), both sides detect that the peer can continue syncing in the same session and start over automatically when needed. Previously the user had to start another sync session manually. To the user this is shown as "number of cycles" in a sync session in the sync report. "Restart" is the process of entering a new cycle. The cycles are also visible in the command line output as multiple INFO lines: [INFO] eds_contact: starting first time sync from client (peer is server) [INFO] creating complete data backup of source eds_contact before sync (enabled with dumpData and needed for prin Local data changes to be applied during synchronization: *** eds_contact *** no changes [INFO] eds_contact: sent 1/1 [INFO] eds_contact: started [INFO] eds_contact: first time sync done successfully [INFO] eds_contact: starting normal sync from client (peer is server) <=== [INFO] eds_contact: started <=== [INFO] eds_contact: normal sync done successfully <=== [INFO] creating complete data backup after sync (enabled with dumpData and needed for printChanges) Synchronization successful. Changes applied during synchronization: +---------------|-----------------------|-----------------------|-CON-+ | | LOCAL | REMOTE | FLI | | Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS | +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ | eds_contact | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | | refresh-from-local, 2 cycles, 0 KB sent by client, 0 KB received | ^^^^^^^^ | item(s) in database backup: 1 before sync, 1 after it | +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ | start Tue Feb 7 17:07:49 2012, duration 0:03min | | synchronization completed successfully | +---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ * SyncEvolution <-> SyncEvolution sync: negotiate UID support via SyncCap (BMC #22783) The semantic of UID/RECURRENCE-ID in calendar data is now tracked per data store involved in a sync. If full iCalendar 2.0 semantic (= IDs are globally unique) is guaranteed, then pairs are found based on these IDs. Otherwise pairs must be found by looking at item attributes. Previously a hack was used to detect this kind of support (any kind of SyncEvolution instance was assumed to support it, although some backends do not). * syncevolution.org packages: fixed D-Bus server autostart in .deb and .rpm packages syncevo-dbus-server wasn't started automatically as part of a user session because /etc/xdg/autostart/syncevo-dbus-server.desktop wasn't included in the packages. This broke auto syncing after a session restart (required manually starting SyncEvolution). * syncevolution.org packages: support KDE The traditional "syncevolution-evolution" package was replaced with "syncevolution-bundle". A meta "syncevolution-evolution" package depends on it, to support seamless updates for users who have "syncevolution-evolution" installed. Binary dependencies of the main .deb are ignored for backends because loading them is optional. The new "syncevolution-kde" package has the right dependencies for KDE/Akonadi, while "syncevolution-evolution" mostly just lists standard libs if the "EDS compatibility" mode is used, where libebook/libecal are loaded dynamically. Platform specific code (GNOME keyring, KDE wallet) was moved into loadable, optional modules, to allow installation of the SyncEvolution bundle without forcing the installation of unused system components. * D-Bus: use GIO D-Bus instead of libdbus if available When compiling from source, the more modern GIO D-Bus is used instead of libdbus if available and recent enough (>= 2.30). syncevolution.org binaries still use libdbus, to stay compatible with older Linux distros. * several minor bug fixes syncevo-dbus-server now runs under valgrind in the nightly testing, plus several more test scenarios were added. This helped to find and fix various minor memory handling issues. * developers: backend API changes beginSync/endSync() (aka m_startDataRead/m_endDataWrite) may now be called multiple times per SyncSource instance life cycle. SyncSources derived from TrackingSyncSource should work without changes. Use the Client::Source::*::testChangesMultiCycles test to check whether your backend supports this correctly. Reading and deleting must throw a 404 status exception when an item is not found. The Client::Source::*::*404 tests cover this. The special semantic of the former RegisterSyncSource::InactiveSource (invalid pointer of value 1) caused bugs, like using it in --print-databases (=> segfault) or not being able to store the result of a createSource() directly in a smart pointer (=> potential leak in SyncSource::createSource()). Obviously a bad idea to start with. Replaced with a RegisterSyncSource::InactiveSource() method which creates a real, inactive SyncSource instance which can and must be deleted by the caller. This is a SyncSource API change for backend developers. Instead of RegisterSyncSource::InactiveSource, return RegisterSyncSource::InactiveSource(). Comparisons against RegisterSyncSource::InactiveSource needs to be replaced with a call to the new SyncSource::isInactive(). Long-running backend calls are encouraged to check for events on the main glib context (either in a loop or with g_main_context_iteration(NULL)) and abort when SuspendFlags::getSuspendFlags().getState() returns SuspendFlags::ABORT. * packagers: libgdbussyncevo is now installed as a normal library in /usr/lib, even though SyncEvolution is the only user. pcrecpp is now a new hard dependency. Upgrading from release 1.2.x: The sync format of existing configurations for Mobical (aka Everdroid) must be updated manually, because the server has encoding problems when using vCard 3.0 (now the default for Evolution contacts): syncevolution --configure \ syncFormat=text/x-vcard \ mobical addressbook The Funambol template explicitly enables usage of the "refresh-from-server" sync mode to avoid getting throttled with 417 'retry later' errors. The same must be added to existing configs manually: syncevolution --configure \ enableRefreshSync=TRUE \ funambol Upgrading from releases before 1.2: Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Releases >= 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. SyncEvolution 1.2.1 -> 1.2.2, 13.01.2012 ======================================== Maintenance release with various bug fixes. * syncevo-dbus-server + ConnMan: fixed "online" detection (BMC #21541, BMC #24587) SyncEvolution did not recognize any cellular connectivity as suitable for syncing. The strict check for certain "connected technology" is unnecessary, anything which makes the computer "online" should be good enough. So now it just uses the ConnMan "State" property. Additional benefit: will continue to work with ConnMan 1.0, which won't have the "ConnectedTechnologies" property anymore. The Bluetooth available check was also (incorrectly) using the ConnMan API. Now asssume that OBEX/Bluetooth is always available. * automatic backups: added INFO messages and fixed dumpData/printChanges (BMC #24619) Point out that backups are created (user might be unaware otherwise and wonder about the delay), explain why (so that users know how to turn it off). Turning these backups off with dumpData=0 printChanges=0 had to be fixed, backups were always written previously. * EDS compatibility: bumped version check for EDS 3.2 SyncEvolution is known to work with EDS 3.2. Therefore use the libebook/ecal/edataserver libs from 3.2 if available, without warnings in the --version output. Also happens with inconsistent distro setups where the old libs are available and would have been prefered by SyncEvolution 1.2.1 even though the old libs no longer work with EDS 3.2. * GTK-UI: do not accept service config without a username (BMC#23106) Instead of creating such a config, an error dialog is shown. * GTK-UI: updated translations * fixed various compile issues, primarily on Fedora Core 17 (unistd.h/ssize_t, invoking syncevolution during compilation, missing src/dbus/qt/configure-sub.in) SyncEvolution 1.2 -> 1.2.1, 25.11.2011 ====================================== Maintenance release with various bug fixes. * GTK UI + config: fix "custom server" setup (BMC #13511) When the "default" config template (= ScheduleWorld) was downgraded to "not consumer ready" in SyncEvolution 1.1.0.99.1, setting up a custom SyncML service in the GTK UI stopped working because the UI wouldn't show the "not consumer ready" config. The problem described above is deterministic and fixed now. Initially the problem seemed to be random. So perhaps there is also another, related issue. * phone sync: delete<->delete conflict + phone calendar+todo sync (BMC #23744) When deleting an item on phone and locally, the next sync failed with ERROR messages about "object not found". Retrying the sync then worked. * Nokia: prevent accidental usage of "calendar" or "todo" sources Nokia phones use a combined "calendar+todo" source for syncing. The "calendar" and "todo" sources also exist because that is where local databases are configured. In such a setup, syncing always has to use "calendar+todo". For example, to refresh from the Linux desktop to the phone, use: --sync refresh-from-server calendar+todo To work with items (restore, show local content), use the underlying sources, as in: --print-items calendar It was possible to accidentally sync with the "calendar". This commit prevents that by adding an invalid URI setting to the "calendar" and "todo" sources in the Nokia and Ovi templates. Existing configs are not touched, so beware when you already have configured your Nokia phone. * vCard: X- chat extensions were limited to one instance per kind For example, only one Jabber account could be synchronized. This was caused by an incomplete definition of the conversion to and from vCard. * syncevo-dbus-server + phone sync: catch SIGPIPE to avoid premature exit Frederik Elwert reported that running a local sync with a phone via Bluetooth caused the syncevo-dbus-server to shut down during a sync. Explicitly telling the process to ignore the SIGPIPE signal solved that problem. * syncevo-http-server: support chained SSL certificates So far, the file pointed to by --certificate-file had to contain the server certificated (signed by a CA known to the client) and (optionally) a client certificate. Now the file may also contain additional intermediate certificates which will be sent to the client (chained certificates). * documentation: added glossary and command line conventions sections, improved listing of properties, embedd property definitions in man page, README and README.html * EDS compatibility: fixed inconsistency in libecal check The check for the _r variants in libical still used an older max version. This might have prevented using them (if not found) or could have led to a mixture of old and new libecal in the same process (probably crashed). * glib: avoid including glib/*.h headers directly Recent glib deprecates the direct inclusion of some of its headers, in favor of including glib.h. Doing that here whenever possible, so perhaps it now compiles on Fedora 17 (untested). SyncEvolution 1.1.1 -> 1.2, 13.10.2011 ====================================== The major new feature of the 1.2 release is support for non-SyncML protocols in general and CalDAV/CardDAV in particular. ActiveSync support is in development and will be in 1.3. These protocols are implemented as backends which are combined with other backends by SyncEvolution in a so called "local sync". The GTK sync-ui does not yet support configuring non-SyncML protocols. See the README.rst and man page for more information on how to use the new feature via the command line. Properties not supported by SyncML servers can now be preserved locally in two-way synchronization (BMC #15030). This depends on information about what properties a SyncML server supports ("CtCap"), which is typically not provided by servers. SyncEvolution contains a copy of that information for Google Contacts (BMC #15029). Akonadi backend and KWallet support were merged. They are not included yet in syncevolution.org binaries. To use them compile from source. The configuration format was updated to solve a conceptual problem inherited with the legacy property names: the "type" property had multiple, sometimes conflicting roles. For example, setting the preferred data format for sync with one peer might have changed the backend selection for some other peer (BMC #1023). Now "backend/databaseFormat/syncFormat/forceSyncFormat" replace "type". "type" is still accepted by the command line as alias. Upgrading from releases before 1.2: Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. Release 1.2 automatically migrates configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. Other changes: * Using the --sync-property and --source-property command line options is optional, just specifying the property assignment is enough. * syncevo-http-server was enhanced considerably. See http://syncevolution.org/wiki/http-server-howto * support NetworkManager API >= 0.9 (BMC #19470) * syncevolution.org binaries: now compatible with Debian Testing/libnotify.so.4 (BMC #22668) libnotify is not linked directly into syncevo-dbus-server in the syncevolution.org binaries. Instead libnotify.so.1 till .so.4 (current Debian Testing) are opened opened dynamically and the necessary functions are looked up via dlsym(). Not finding the libraries or the functions silently disables this notification mechanism. * Sync mode is recorded when running in SyncML server mode (BMC #2786). * syncevo-dbus-server automatically stops when some of its libraries are updated and restarts if auto-syncing is on (BMC #14955). * Added code for Buteo, mKCal and QtContacts in MeeGo. Buteo and mKCal were removed again from MeeGo, so the code is obsolete. The QtContacts backend may be still be useful to access items via that API, but for syncing on MeeGo the normal EDS backend is used since MeeGo reverted back to EDS as PIM storage. * "databasePassword" source property: lookup failure in keyring (BMC #22937) The databasePassword also wasn't looked up at all when doing item operations via the command line. When configuring sources for an HTTP server, the config name typically is just the context (@foo). When using the config in the HTTP server, the config name is the peer inside that context (client@foo). Because the GNOME keyring lookup keys for the "databasePassword" (more specifically, the object name) contained the full config name which was different in both cases, looking up the saved password failed. The solution is to normalize the config name (to accomodate for different ways of spelling it) and use only the context, with @ as before. This will break existing setups where the object name in the keyring (incorrectly) includes the full config name. In that case just configure the source again to set the password anew. * Evolution Calendar: fixed detached recurrence support (BMC #22940) When manipulating a meeting series with more than one detached recurrence certain sequences of operations could incorrectly fail with "UID already exists". * iCalendar 2.0: must set VALUE in EXDATE (part of BMC #22940) EXDATE has a VALUE parameter, which wasn't defined in the XML profile. Didn't seem to matter at all in practice, but wasn't standard-compliant. * GTK sync-ui: wrap sync service descriptions (BMC #7199) Descriptions of different sync services are not fully visible unless word-wrapping gets enabled. * CalDAV/CardDAV + local storage: avoid empty properties The main motivation for this change is that a recent Apple Calendar server rejects vCards with empty BDAY property. Another reason is that keeping the data as small as possible is desirable by itself. Sending an empty property serves as a hint for the peer that the property is supported. This is not necessary when storing an item in a backend. Therefore this commit disables empty properties for all backends which do not themselves set the m_backendRule Synthesis info value. * Google Contacts: ensure that first/middle/name are set when storing in EDS (BMC #20864) Evolution and the MeeGo UX assume that first/middle/last name are set. That is not the case when a contact is created in the Google Contacts web interface. Such contacts are sent by Google without the N property. SyncEvolution now tries to recreate the name components from the FN string, by splitting at word boundaries and assuming " " or ", " format. Obviously this heuristic fails for some locales. * Evolution Calendar: fixed error handling for broken TZIDs * Sony Ericsson: use ISO-8859-1 for all devices (BMC #14414) Passing invalid UTF-8 strings into libecal caused glib to abort syncevo-dbus-server. * auto sync: show all failed syncs except for temporary network errors (BMC #21888) Notifications were meant to be shown for all errors except temporary ones. This has never been implemented correctly since the feature was introduced: instead of hiding known temporary errors, all errors except 500 (fatal error) were suppressed. * vCard: inline local photo data (BMC #19661) Some platforms (Maemo, MeeGo) store photos in separate files. Now SyncEvolution efficiently includes that photo data in the generated vCard right before sending it to a peer; previously it sent a useless local file:// URI. The Maemo port has a less efficient workaround for that which now should be obsolete. * syncevo-dbus-server: online status wrong without Network Manager or ConnMan (BMC #21543) When neither Network Manager nor ConnMan are running, network presence was "not online". This prevented running automatic syncs. For developers: * modified backend API - ClientTestConfig modernized - InsertItemResult::m_merged turned from boolean to enum * testing and compilation changes; for example, the minimum version of libsynthesis is now checked at configure time instead of failing at runtime due to missing features in the Synthesis engine SyncEvolution 1.1.99.7 -> 1.2, 13.10.2011 ========================================= Some more bug fixes and testing improvements. * fixed potential invalid memory access in add<->add conflict handling * fixed memory leak in workaround for EDS bug * CalDAV/CardDAV: handle ETags without quotation marks (eGroupware) * updated README: warning about sync direction moved to --sync option SyncEvolution 1.1.99.6 -> 1.1.99.7, 15.09.2011 ============================================== Mostly bug fixes again. Some are a bit more intrusive, thus another pre-release. * syncevolution.org binaries: now compatible with Debian Testing/libnotify.so.4 (BMC #22668) libnotify is not linked directly into syncevo-dbus-server in the syncevolution.org binaries. Instead libnotify.so.1 till .so.4 (current Debian Testing) are opened opened dynamically and the necessary functions are looked up via dlsym(). Not finding the libraries or the functions silently disables this notification mechanism. * calendar sync: better handling for add<->add conflicts (partly fixes BMC #22783) When both sides of a sync have added the same event, the sync must determine which one is more recent instead of blindly overwriting always the same side. Such conflicts are typically rare except for enterprise scenarios where meeting invitiations are processed automatically by a groupware (Exchange, Google Calendar/Mail, ...) and then the attendee status is updated on one side. SyncEvolution now does the necessary age comparison and preserves the more recent data for most properties. In some properties the data from both sides is preserved by concatenating the text (description, location, ...). It remains to be seen whether that is really desirable. Also, sync statistics are slightly off: the incoming item is counted as "added" even though it gets turned into an update. * item operations: authentication problem for WebDAV when using keyring (BMC #21311) The password still wasn't looked up in the keyring when using --import/export/delete-items. * "databasePassword" source property: lookup failure in keyring (BMC #22937) The databasePassword also wasn't looked up at all when doing item operations via the command line. When configuring sources for an HTTP server, the config name typically is just the context (@foo). When using the config in the HTTP server, the config name is the peer inside that context (client@foo). Because the GNOME keyring lookup keys for the "databasePassword" (more specifically, the object name) contained the full config name which was different in both cases, looking up the saved password failed. The solution is to normalize the config name (to accomodate for different ways of spelling it) and use only the context, with @ as before. This will break existing setups where the object name in the keyring (incorrectly) includes the full config name. In that case just configure the source again to set the password anew. * Evolution Calendar: fixed detached recurrence support (BMC #22940) When manipulating a meeting series with more than one detached recurrence certain sequences of operations could incorrectly fail with "UID already exists". * iCalendar 2.0: must set VALUE in EXDATE (part of BMC #22940) EXDATE has a VALUE parameter, which wasn't defined in the XML profile. Didn't seem to matter at all in practice, but wasn't standard-compliant. * GTK sync-ui: wrap sync service descriptions (BMC #7199) Descriptions of different sync services are not fully visible unless word-wrapping gets enabled. * source configs: don't check "backend" unless it is needed When using a config which has sources with a backend type set which is not currently available, an error was thrown even if those sources weren't even part of the current operation (for example, syncing another source which is currently supported). * config migration: avoid name conflicts and auto syncing of old configs (BMC #22691) When (auto-)migrating a config, it was possible that a name for the peer, say foo.old, was chosen for the renamed config although there was already such a config, for example foo.old in ~/.sync4j. Besides being confusing for users, this also led to a bug in the code where it copied from the older config with the foo.old name. The main problem fixed is the disabling of auto syncing in the old config. Otherwise it was still used by syncevo-dbus-server for syncing, which triggered another auto-migration, ad infinitum... * auto syncing: must check whether enabled when looking at unknown URLs (part of BMC #22691) "syncURL = insert your URL here" with "autoSync = 0" did lead to auto sync attempts although it wasn't enabled. A check for "auto syncing enabled" was missing for the "unknown transport" case. * CalDAV/CardDAV + local storage: avoid empty properties The main motivation for this change is that a recent Apple Calendar server rejects vCards with empty BDAY property. Another reason is that keeping the data as small as possible is desirable by itself. Sending an empty property serves as a hint for the peer that the property is supported. This is not necessary when storing an item in a backend. Therefore this commit disables empty properties for all backends which do not themselves set the m_backendRule Synthesis info value. * Apple CardDAV: apply PHOTO import/export scripts by default A recent Apple Calendar server (correctly) rejects the invalid PHOTO;TYPE=unknown: property in a vCard. This internal representation must be cleared before serializing the field list. * for developers: modified backend API - ClientTestConfig modernized - InsertItemResult::m_merged turned from boolean to enum * testing and compilation changes; for example, the minimum version of libsynthesis is now checked at configure time instead of failing at runtime due to missing features in the Synthesis engine SyncEvolution 1.1.99.5 -> 1.1.99.6, 17.08.2011 ============================================== Mostly bug fixes, some improvements in testing and packaging. This release was tested successfully with DAViCal 0.9.9.4. * CalDAV: fixed incorrect change tracking causing "event not found" (BMC #22329) * CalDAV: handle delete<->delete conflict during local sync (BMC #22327) If the same event was deleted both locally and in the CalDAV server, syncing failed with "event not found". * Google Contacts: ensure that first/middle/name are set when storing in EDS (BMC #20864) Evolution and the MeeGo UX assume that first/middle/last name are set. That is not the case when a contact is created in the Google Contacts web interface. Such contacts are sent by Google without the N property. SyncEvolution now tries to recreate the name components from the FN string, by splitting at word boundaries and assuming " " or ", " format. Obviously this heuristic fails for some locales. * CalDAV: continue despite Google Calendar access problems (see BMC #19484) An attempt to work around "403 You don't have access to change that event" errors, perhaps caused by http://code.google.com/p/google-caldav-issues/issues/detail?id=38 The problem is now recorded instead of aborting the sync. The sync then ends in a 22001 = "partial failure" error and the operation will be retried in the next sync. * CalDAV: transform UTC RECURRENCE-ID for Evolution (BMC #22594) Evolution showed a meeting twice on the day of a modified recurrence, if the meeting series was originally created and modified in Exchange, then imported into Google Calendar. * CalDAV syncevolution.org binaries now works when libneon.so.27 or libneon-gnutls.so.27 (Debian) are installed. Previously libneon.so.27 was required, which is no longer available in Debian Testing. * syncevo-dbus-server/gdbus: fixed segfault when asked for properties when none are available (BMC #22152) * Evolution Calendar: fixed error handling for broken TZIDs * Sony Ericsson: use ISO-8859-1 for all devices (BMC #14414) Passing invalid UTF-8 strings into libecal caused glib to abort syncevo-dbus-server. * item operations: authentication problem for WebDAV when using keyring (BMC #21311) The password wasn't looked up in the keyring when using --print-items/import/export/... * WebDAV: fixed item operations without configuration (BMC #22164) Previously failed with "[ERROR] : virtual read-only configuration node, cannot write property webDAVCredentialsOkay = 1". * auto sync: show all failed syncs except for temporary network errors (BMC #21888) Notifications were meant to be shown for all errors except temporary ones. This has never been implemented correctly since the feature was introduced: instead of hiding known temporary errors, all errors except 500 (fatal error) were suppressed. * vCard: inline local photo data (BMC #19661) Some platforms (Maemo, MeeGo) store photos in separate files. Now SyncEvolution efficiently includes that photo data in the generated vCard right before sending it to a peer; previously it sent a useless local file:// URI. The Maemo port has a less efficient workaround for that which now should be obsolete. * syncevo-dbus-server: online status wrong without Network Manager or ConnMan (BMC #21543) When neither Network Manager nor ConnMan are running, network presence was "not online". This prevented running automatic syncs. * fixed compile issues with Debian Testing/gcc 4.6.1 Known issues, might still be resolved for the final 1.2: -------------------------------------------------------- * syncevolution.org binaries: libnotify1 -> libnotify4 incompatibility (BMC #22668) Newer distros no longer have the libnotify.so.1 that syncevolution.org binaries depend on. As a workaround it is possible to install the libnotify1 package from older distro releases. * CalDAV: add<->add conflicts (BMC #22669) Suppose the same meeting invitation for event UID=FOO is processed in both Evolution and Google Calendar. This always happens when the meeting invitation emails is sent to Google Mail, then later viewed in Evolution. On the Evolution side, the invitation is accepted. In Google Calendar this is still open. When syncing in that state the sync engine does not recognize that both sides have added the same meeting and the "meeting accepted" information eventually gets lost. As a workaround, always synchronize the calendar before processing meeting invitation emails. SyncEvolution 1.1.99.1 -> 1.1.99.5, 13.07.2011 ============================================== Release 1.1.99.5 is the first release candidate for 1.2. It has gone through a long stabilization period and thus is suitable for normal users. The major new feature of the 1.2 release is support for non-SyncML protocols in general and CalDAV/CardDAV in particular. ActiveSync support is in development. These protocols are implemented as backends which are combined with other backends by SyncEvolution in a so called "local sync". The GTK sync-ui does not yet support configuring non-SyncML protocols. See the README.rst and man page for more information on how to use the new feature via the command line. Properties not supported by SyncML servers can now be preserved locally in two-way synchronization (BMC #15030). This depends on information about what properties a SyncML server supports ("CtCap"), which is typically not provided by servers. SyncEvolution contains a copy of that information for Google Contacts (BMC #15029). Akonadi backend and KWallet support were merged. They are not included yet in syncevolution.org binaries. To use them compile from source. The configuration format was updated to solve a conceptual problem inherited with the legacy property names: the "type" property had multiple, sometimes conflicting roles. For example, setting the preferred data format for sync with one peer might have changed the backend selection for some other peer (BMC #1023). Now "backend/databaseFormat/syncFormat/forceSyncFormat" replace "type". "type" is still accepted by the command line as alias. Old configurations can still be read. But writing, as it happens during a sync, must migrate the configuration first. In contrast to earlier, more experimental releases in the 1.2 series, 1.1.99.5 and later automatically migrate configurations. The old configurations will still be available (see "syncevolution --print-configs") but must be renamed manually to use them again under their original names with older SyncEvolution releases. Other changes: * syncevo-http-server was enhanced considerably. See http://syncevolution.org/wiki/http-server-howto * support NetworkManager API >= 0.9 (BMC #19470) * Sync mode is recorded when running in SyncML server mode (BMC #2786). * syncevo-dbus-server automatically stops when some of its libraries are updated and restarts if auto-syncing is on (BMC #14955). * Using the --sync-property and --source-property command line options is optional, just specifying the property assignment is enough. * Added support for Buteo, mKCal and QtContacts in MeeGo. Buteo and mKCal were removed again from MeeGo, so the code is obsolete. The QtContacts backend may be still be useful to access items via that API, but for syncing on MeeGo the normal EDS backend is used since MeeGo reverted back to EDS as PIM storage. * code cleanup and various minor fixes/improvements, see ChangeLog SyncEvolution 1.1 -> 1.1.1, 26.12.2010 ====================================== Maintenance release, in particular improving syncing with phones. There was a bug that could cause all kinds of weird behavior after a failed sync with a phone, so updating is highly recommended. * Synthesis engine: fixed a corruption issue in internal meta data which caused duplicates and other problems in a pretty indeterminstic way; apparently caused by failed syncs (BMC #11044). * Synthesis engine: recurrence rules with end date now sent correctly to phones (BMC #11241). The RRULE property was not encoded correctly previously during the iCalendar 2.0 -> vCalendar 1.0 conversion. Events with recurrence count were okay. Probably also affected SyncML servers without iCalendar 2.0 support. The fix was confirmed to work with Nokia phones. It also helps with Sony Ericsson phones, but at least the t700 still has a problem: depending on the phone's time zone, it repeats the event for one day too long (BMC #10092). * Synthesis engine: fixed broken time zone information when sending to phone; previously that broke sending calendar updates to Nokia phones (BMC #9600). iCalendar 2.0 time zone definitions imported from libical were not encoded correctly in vCalendar 1.0 items as sent to phones. Nokia phones accepted such data when part of a new event, but rejected updates of it. * Synthesis engine: shorter TZIDs, might help N900 calendar (BMC #6680). The shorter TZIDs will be included in iCalendar 2.0 data exported by libsyntesis and thus SyncEvolution. This change is motivated primarily by the observation that the N900 calendar storage can handle TZID=, but not TZID=/softwarestudio.org/Tzfile/. * ScheduleWorld: disable configuration template because service has shut down. The template is only hidden from the GTK sync-ui, but remains in SyncEvolution for the time being because it is referenced in several places. * Evolution CalDAV: added workaround for "must sync twice" (BMC #10265) The Evolution CalDAV backend seems to update its data when closing the database, not when opening it. As a result, syncevolution had to be run twice to see all data changes. The workaround is to open the database twice at the start of the sync. This is done for all calendar databases, regardless of which backend they use, in case that some other (yet unknown) backend needs the same workaround. * GTK sync-ui: workaround for "Sync Now" button not reacting to online status changed (BMC #9949). * Changed slow sync handling. Some users have complained about getting duplicated contacts (BMC #10081). The exact reason is not known (no useful logs provided yet), but it might be due to using "duplicate" as resolution strategy during slow syncs. This caused slightly different contacts to be duplicated instead of merging the two copies, reasoning that "no data loss" is better than "duplicates". This release switches to a mode where the engine tries harder to avoid duplicates by merging data if modification time stamps are available for contacts (usually they are). When fields differ, the more recent data is kept. * convert absolute alarm back to relative (BMC #11233) Experiments show that at least Nokia phones (and thus perhaps also Mobical.com) interpret a fixed alarm as "repeat alarm with the same relative offset as on first occurrence". The same transformation to relative alarm times is applied whenever the transformation to absolute alarm is enabled for a peer. * Sony Ericsson: enable conversion to absolute alarm times (BMC #10092) Like Nokia and Mobical.net, Sony Ericsson phones also seem to be unable to deal with relative alarm times - verified with t700. * Sony Ericsson C510: workaround for SyncML violation The phone does not sent identifiers for the target database; using the source identifier as fallback allows a sync to run. * Fixed a regression affecting users who had created a config with SyncEvolution < 1.0. Using the config worked once, then failed with "No configuration for ... found". Users must manually remove the empty "peers" directory inside their affected configuration, the fix only makes configs without that directory usable again (BMC #9381). * Removed obsolete workaround for older mKCal calendar storage. * Fixed error message in QtContacts backend. * Same SYNCEVOLUTION_DEBUG code as in master branch. * Some updates to synccompare, including a workaround for a Perl bug seen on Debian Testing with Perl 5.10.1-16 (Perl panic). * Fix compilation of syncevo-dbus-server with libnotify 0.7.0 (BMC #10453). * Fixed compilation on Debian GNU/Hurd (no MAX_PATH, Mac OS X confusion). SyncEvolution 1.0.1 -> 1.1, 26.10.2010 ====================================== An incremental update, resolving issues where the fixes would have been too intrusive for a 1.0.x release. In particular compatibility with Nokia phones was improved. Some new features were also included (command line options for manipulating items, backends for MeeGo PIM storages). Details: * bug fix in sync-ui: wrong direction of one-way data transfers with devices (BMC #7091) * bug fix in syncevo-dbus-server: incorrect Presence status after config change (BMC #8453) Shows up in sync-ui as "'Sync Now' button active after creating a config while offline". * sync-ui (GTK version): app is now listed as "SyncEvolution (GTK)" under "Office" * Nokia phones: avoid data loss in two-way sync due to X-EVOLUTION-UI-SLOT (BMC #2566) * Nokia phones: alarm times in UTC, sending PHOTO (BMC #1657, #5860) * included all phone templates submitted to syncevolution.org Wiki (BMC #5727) * syncevo-phone-config: set consumerReady in output, more useful for Wiki (BMC #3803) * workaround for D-Bus timeouts in EDS libecal/libebook (BMC #4026) * added generic command line options for importing, exporting, updating, listing and deleting items in the different backends (http://syncevolution.org/blogs/pohly/2010/manipulate-evolution-kcalextendedmkcal-qtcontacts-pim-items-uniform-command-line) * added backends for mKCal and QtContacts (MeeGo PIM storage), meant to be used for manipulating this data on the command line * enhanced D-Bus interface (BMC #3558, #3559, #3560, #3562, #3563, #7761, #7766) * the command line tool now warns when running against a different D-Bus daemon (BMC #3563) * creating and configuring sources in a context (without peer-specific properties) is now supported * improved documentation: README.rst, man page, and --help output * fixed some compile issues (BMC #6367), improved nightly testing SyncEvolution 1.0 -> 1.0.1, 16.07.2010 ====================================== A bug fix release. The main reason for releasing it is that SyncEvolution 1.0 no longer worked on recent distros (Fedora Core 13, Debian testing) because of a name clash between the Bluez D-Bus utility code and recent glib. Details: * compile fix for FC 13 (and possibly others): use private copy of gdbus (BMC #3556) * sync-ui: prevent overwriting device configs by accident (BMC #3566,1194) Setting up a phone used the template name as config name and overwrote an existing configuration of another phone that was created using that same template. Now the code uses the Bluetooth device name as set on the device and checks for (less likely) collisions. It also sanitizes the name to avoid complicated config names (only relevant when also using the command line). * syncevo-dbus-server: accept 'application/vnd.syncml+xml; charset=UTF-8' for starting an HTTP session (BMC #3554) The redundant charset specification was set by the Funambol Thunderbird client. Because of a literal comparison against 'application/vnd.syncml+xml' the messages were rejected. * config fix: operations on non-peer configs failed (BMC #3157) When running operations on a non-peer configuration (like --restore @default addressbook), the operation fails with [ERROR] : type 'select backend' not supported * ZYB.com: service goes away end of June 2010, template removed (BMC #3310) * some build (BMC #2586, BMC #3557) and language updates SyncEvolution 0.9.2 -> 1.0, 11.06.2010 ====================================== Major new features compared to previous stable release: * synchronize directly with a phone over Bluetooth/OBEX * accept Bluetooth/OBEX connections in cooperation with obexd >= 0.19 * run SyncEvolution as a rudimentary HTTP SyncML server The GTK sync-UI can be used to select a paired phone and create a configuration for it based on the bundled configuration templates. Configuration templates are included for Nokia phones; for other phones see the http://syncevolution.org/development/sync-phone HOWTO and check out the Wiki there. Some users have already reported success for Sony Ericsson phones and added setup instructions. New templates from the Wiki can be dropped into ~/.config/syncevolution-templates under an arbitrary file name. Unexpected slow syncs can be detected when running as client (MB #2416) and unless turned off (see "preventSlowSync"), SyncEvolution aborts the session so that the situation can be analyzed. A refresh from client or server might be more suitable. The command line tool provides instructions at the end of its output. The GTK sync-UI points towards its recovery dialog. Automatic synchronization is supported by the syncevo-dbus-server (MB #6378). When that is installed, it will be started as part of a user session and keep running to trigger syncs in the background. Notifications are emitted when syncs start, end or fail (MB #10000). Automatic synchronization can be enabled separately for each peer ("autoSync=0/1", off by default), will be done at regular intervals ("autoSyncInterval=30" minutes) when online long enough ("autoSyncDelay=5" minutes). That last option ensures that a) an automatic sync does not attempt to use a network connection unless it was already active and b) hopefully is also around long enough to complete the sync. The Synthesis XML configuration was split up into different parts which are assembled from /usr/share/syncevolution/xml. Files in ~/.config/syncevolution-xml override and extend the default files, which my be useful when adding support for a new phone. SyncML servers: * ZYB.com now works thanks to a workaround for anchor handling (MB #2424); only contacts tested because everything else is considered legacy by ZYB.com * Horde: avoid confusing the server with a deviceId that starts like the ones used in old Funambol clients, helps with calendar sync (MB #9347) * Mobical.net (and other, similar services): fix vCalendar 1.0 alarm properties before importing them (MB #10458) * desknow.com works when switching to SyncMLVersion = 1.1 * Funambol, Memotoo (and probably others): preserve meeting series when receiving update for detached recurrence (BMC #1916) Evolution: * addressbook backend: avoid picking CouchDB, second try (MB #7877) * calendar backend: minor fix for change tracking when deleting a single instance of a recurring event * workaround for Evolution 2.30: "timezone cannot be retrieved because it doesn't exist" is triggered incorrectly when importing non-standard timezone definitions because libecal changed an error code (MB #9820) Performance and reliability improvements (MB #7708): * synccompare much faster * database dumps consume less disk space * more intelligent about expiring obsolete session directories and backups * database accesses are reduced in several backends * shorter logs (MB #8092) * message resending helps under unreliable network connectivity ("RetryInterval") * full support for suspend&resume in SyncEvolution client to SyncEvolution or Synthesis server syncs * better handling of certain third-party time zone definitions (BMC #1332) Improved GTK sync-UI: * revised config screen: all in one list where entries can be expanded, integrated setup of sync with other devices * recovery support: restore from backup, unexpected slow sync handling * spinner while network is in use (MB #2229) * interactive password requests (MB #6376) * uses new D-Bus API Command line: * fixed printing of rejected items (MB #7755) * consistent logging of added/updated/deleted items with short description * improved error reporting (textual descriptions instead of plain error codes MB #2069, partial success MB #7755, record and show first ERROR encountered MB #7708) * can create new sources (MB #8424) * runs operations inside daemon and thus avoids conflicts with operations done by other clients; for testing purposes (like running a client which talks to a local server in the daemon) it is still possible to ignore the daemon (--daemon=no, MB #5043) * revised README, now also available as man page (BMC #690) Redesigned and reimplemented D-Bus API, used by sync-UI and command line: * central syncevo-dbus-server controls configurations and sync sessions: http://syncevolution.org/development/direct-synchronization-aka-syncml-server * accepts incoming SyncML connection requests and messages received by independent transport stubs (obexd, HTTP server, ...) * can be used by multiple user interfaces at once * fully documented, see src/dbus/interfaces and http://api.syncevolution.org * no longer depends on dbus-glib with hand-written glue code for C++, instead uses gdbus plus automatic C++ binding generated via C++ templates Revised configuration layout (MB #8048, design document at http://syncevolution.org/development/configuration-handling): * several peer-independent sync and source properties are shared between multiple peers * they can be accessed without selecting a specific peer, by using an empty config name or with the new "@" syntax * user interface of command line unchanged * old configurations can be read and written, without causing unwanted slow syncs when moving between stable and unstable SyncEvolution versions * old configurations can be migrated with the "--migrate" command line switch; however, then older SyncEvolution can no longer access them and migrating more than one old configuration causes the second or later configuration to loose its "deviceId" property (which is shared now), causing a slow sync once * config names may contain characters that are not allowed in the file names used for the underlying files; will be replaced with underscores automatically (MB #8350) Upgrading from 0.9.x: * Upgrading and downgrading should work seamlessly when using existing configurations. * The new configuration layout is only used when creating new configurations or explicitly invoking "syncevolution --migrate" (see above). Such configs cannot be used by older SyncEvolution releases. * The new "RetryInterval" property causes messages to be resent after 2 minutes (increased from 1 minute in previous 1.0 betas). At least the Funambol server is known to not handle this correctly in all cases (http://funzilla.funambol.com/show_bug.cgi?id=7910). So in the Funambol config template the interval is set to zero, disabling the feature. Disabling the feature must be done manually in existing Funambol configurations. SyncEvolution 1.0 beta 3 -> 1.0 final, 11.06.2010 ================================================= Bug fixes and new features: * Configuration templates are stored in a single file (BMC #1208). New templates (like something downloaded from http://syncevolution.org/wiki) can be dropped into $HOME/.config/syncevolution-templates using an arbitrary file name. * Progress and per-source status are now also reported and recorded when running in server mode (BMC #1359). There are still several limitations (sync mode not reported, no information about sent/received/processed items while the sync runs, see BMC #2786). * Better handling of certain third-party time zone definitions (BMC #1332). Better logging to track down such problems. * D-Bus server + command line: return error code when failed (BMC #2193) * syncevo-phone-config: simplified command line options, several bug fixes (syntax error, incorrect handling of calendar+todo, BMC #1197) * Revised README, now also available as man page (BMC #690). Conversion of D-Bus API documentation into .html page (BMC #1745). * Funambol, Memotoo (and probably others): preserve meeting series when receiving update for detached recurrence (BMC #1916) * Fix for potential out-of-bounds memory access (BMC #1007). * HTTP server: fix for potential crash when second session was requested while an older one was still running, initial sync was done without libical time zone information and thus may have mismatched times (BMC #2435) * Nokia E55: convert alarm times (BMC #1657). This is done via a new remote rule in /usr/share/syncevolution/xml/remoterules/server/46_E55.xml If another phone needs the same treatment, then copy that file to ~/.config/syncevolution-xml/remoterules/server and edit the element. * GTK GUI: styling fix (BMC #1372), updated toolbar for MeeGo 1.0 (BMC #1970), avoid duplicating configs when selecting a config created by syncevo-phone-config or the command line (BMC #1266), scroll bars for emergency window (BMC #1296), avoid compile problem on Fedora Core 13 due to name collision with system sync() call, updated translations. SyncEvolution 1.0 beta 2 -> beta 3, 20.04.2010 ============================================== One more step towards the long awaited 1.0. 0.1 was released over four years ago and the 1.0 cycle started some time last summer. Beta 3 is considered feature complete at this point. Automatic synchronization is supported by the syncevo-dbus-server (MB #6378). When that is installed, it will be started as part of a user session and keep running to trigger syncs in the background. Notifications are emitted when syncs start, end or fail (MB #10000). Automatic synchronization can be enabled separately for each peer ("autoSync=0/1", off by default), will be done at regular intervals ("autoSyncInterval=30" minutes) when online long enough ("autoSyncDelay=5" minutes). That last option ensures that a) an automatic sync does not attempt to use a network connection unless it was already active and b) hopefully is also around long enough to complete the sync. Detecting online status depends on ConnMan. Without it, SyncEvolution assumes that the network is available. For Bluetooth it is enough to have a peer paired. When SyncEvolution is compiled with a backend sync daemon ("syncevo-dbus-server"), then conceptually that daemon controls the configuration and coordinates manually and automatically started sync sessions. Previously, the command line tool bypassed the daemon by running operations itself. Now it can hand over the command line parameters to the daemon to be executed there ("--daemon=yes", the default if the daemon is available; MB #5043). Command line parameters and output of "syncevolution" are the same as before. Note that the daemon only runs one operation at a time, which delays the command line client when the daemon is busy. For testing purposes (like running a client which talks to a local server in the daemon) it is still possible to ignore the daemon (--daemon=no). Thanks to fixes and improvements in both Synthesis engine and SyncEvolution, suspend and resume are fully supported in client and server (MB #2425). Previously it failed in some cases, as mercilessly exposed by our automated testing. Now all of these tests pass. The HTTP server now also handles message resends by clients correctly. Direct synchronization with older phones (like Sony Ericsson K750i) can be started now by switching to an older version of the SyncML standard ("SyncMLVersion" property, MB #9312). No further interoperability testing with such phones has been done at this time. When acting as client, that same property allows talking to older SyncML servers, like desknow.com. A minor workaround and the right configuration make it possible to synchronize with Nokia N85 and probably also other S60 devices. Added a template for "Nokia S60". Also made the template for "Nokia N900" accessible in the GTK GUI. Because determining which configuration works for a phone involves a lot of trial-and-error, the new "syncevo-phone-config" script automates that process. Other changes: * Mobical.net (and other, similar services): fix vCalendar 1.0 alarm specifications before importing them (MB #10458) * Nokia N900: added a config template for it and disabled the redundant RespURI when using Bluetooth. Preliminary testing shows that this solves some of the issues seen before (MB #10224). * workaround for Evolution 2.30: "timezone cannot be retrieved because it doesn't exist" is triggered incorrectly when importing non-standard timezone definitions because libecal change an error code (MB #9820) * "syncevo-http-server" HTTP server script is included in normal install * syncevolution.org binaries: finally solved the libbluetooth3 incompatibility (MB #9289). Binaries of beta 2 crashed on more recent distros because of that. * SyncML client and Bluetooth: a mobile device running SyncEvolution creates a configuration automatically (MB #6175). The peer contacting us has to use the standard SyncEvolution URIs (addressbook, calendar, todo, memo). * command line: when dealing with the shared non-peer part of a config, it checks for properties which are unsuitable only prints those (MB #8048) * GTK GUI: improved setup of devices, automatic sync switch, some fixes for crashes and other tweaks * Nokia 7210c: send time as UTC instead of relying on time zone information (MB #9907). * command line: setting up a configuration for a "SyncEvolution" server on a client was not possible because the "SyncEvolutionClient" configuration was picked instead (MB #10004). The latter has to be used when configuring a SyncEvolution server to talk to a SyncEvolution client. * restore: no longer updates the time of the backup (MB #9963) * various minor improvements and fixes, see ChangeLog Upgrading: * The new "RetryInterval" property causes messages to be resent after 2 minutes (increased from 1 minute in previous 1.0 betas). At least the Funambol server is known to not handle this correctly in all cases (http://funzilla.funambol.com/show_bug.cgi?id=7910). So in the Funambol config template the interval is set to zero, disabling the feature. Enabling or disabling the feature must be done manually in existing configurations. SyncEvolution 1.0 beta 1 -> beta 2, 23.02.2010 ============================================== Several new features and some bug fixing. Despite some open issues (see below), this release is ready for getting packaged in staging areas of distros as replacement for 0.9.2. As before, documentation for 1.0 is only available in the "Development" section of syncevolution.org, including HOWTOs for setting up the HTTP SyncML server and phones manually. Setting up a phone became a bit easier with beta 2, because SyncEvolution is now integrated with the GNOME Bluetooth panel: once a device with SyncML client support is paired, a button offers to bring up the sync-UI and configure or synchronize with that device. We do a fuzzy match against the Bluetooth device name to find a suitable template (not manufacturer/model, because that is not readily available). Still not many (read: hardly any) templates available, though. The binaries on syncevolution.org are compiled with Bluetooth support. libbluetooth2 or libbluetooth3 should be installed, but are not essential. If there is no suitable version of it, the Bluetooth channel has to be selected manually as part of the syncURL. Unexpected slow syncs are prevented by default, in contrast to beta 1 where this feature was available but turned off. When an unexpected slow sync is detected in a client, users have to follow the instructions provided by the command line or sync-ui and choose how to proceed (explicitly request slow sync, refresh from server or client, restore from backup). SyncEvolution as server currently cannot prevent slow syncs, even when initiating the sync with a phone. In preparation for syncing automatically, logdir and database handling was improved considerably. Backups use less disk space because identical files share the same file content via hard links. This also speeds up the synccompare Perl script. Database dumps and the corresponding comparison are delayed until the session really runs, which avoids doing needless work a) when the server a client tries to contact is unreachable or down and b) by only including sources that are really in use during a sync on the server side. The Synthesis XML configuration was split up into different parts which are assembled from /usr/share/syncevolution/xml. Files in ~/.config/syncevolution-xml override and extend the default files, which my be useful when adding support for a new phone. Summary of changes since 1.0 beta 1: * sync-ui: recovery dialog (MB #8050), device setup, config usable with long strings (MB #9278), fixed displaying of source phases during sync (MB #9320) * sync-ui + syncevo-dbus-server: integration with Bluez to detect paired devices (MB #9216, MB #7089), select template based on device name (MB #7838), detect network and Bluetooth connectivity (only with ConnMan, MB #7700), passwords stored in GNOME keyring by syncevo-dbus-server are shown with dots in sync-ui (MB #9169) * Evolution addressbook backend: avoid picking CouchDB, second try (MB #7877) * Evolution calendar backend: minor fix for change tracking when deleting a single instance of a recurring event * build fixes: Bluetooth compatibility (MB #9289), use libical _r variant of calls because 0.43 has issues in the normal version, conflict with system libsynthesis and libsmltk (MB #9811) * Horde: avoid confusing the server with a deviceId that starts like the ones used in old Funambol clients, helps with calendar sync (MB #9347) * better reporting when SyncEvolution dies during a sync (only happend once when it wasn't installed properly, but still... MB #9844) * performance improvements: synccompare much faster/database dumps consume less disk space/more intelligent about expiring obsolete session directories and backups/database accesses are reduced in several backends (MB #7708), shorter logs (MB #8092) * slow sync detection: now also works in the case where the client detects an anchor mismatch and enabled by default (MB #2416) * OBEX transport: some error handling changes and removal of polling, now also possible via sync-ui + syncevo-dbus-server (MB #9436) * API changes: SyncSource introduces an "isEmpty" operation which is needed for the slow sync detection * SyncML: split up configuration (MB #7712), increased default message size because the old one might have been too small for large DevInf structures * several fixes for virtual data sources ("calendar+todo"): now works on client side, fixed naming on server (MB #9664), fixed error message for slow sync detection, supported in combination with sync-UI (MB #9535) * fixes for shared configuration layout: finding sessions of peers in non-default context, adding sources affected peers in the same context (MB #9329), wrong context during --configure when using shortcut for peers in non-default context (MB #9338) Known gaps for 1.0 final and beyond: Redesigned and reimplemented D-Bus API, required by sync-UI: - 'syncevolution' command line tool bypasses D-Bus server and runs sync sessions itself (MB #5043) - availability of peers not detected when using NetworkManager (connected for HTTP, paired for Bluetooth; MB #7700) SyncML server in general: - suspend/resume support is untested (MB #2425) - the progress events and statistics reported for a SyncML client are not generated when running as SyncML server, will require a fair amount of refactoring in the Synthesis engine (MB #7709) HTTP SyncML server: - a configuration must be created for each peer manually, including a remoteDeviceId value that contains the peer's SyncML device ID (MB #7838) OBEX SyncML server ("sync with phones"): - does not support phones which require a SAN 1.0 message (MB #9312) - determining a working configuration for an unknown phone requires a bit of experimenting, which should be automated (MB #9862) OBEX SyncML client: - parsing of SAN message is rudimentary and depends on an existing local configuration, needs to be refined depending on which SyncML server software it is meant to work with (MB #6175) Automatic sync (MB #6378): - no support for the various server push notification mechanisms - no intelligent detection of local changes - no regular background sync, development is in progress Upgrading from 1.0 beta 1: moving back and forth should work seamlessly Upgrading from 0.9.x: see under beta 1 SyncEvolution 0.9.2 -> 1.0 beta 1, 26.01.2010 ============================================== Compared to the current stable release, 0.9.2, this beta release can also: * synchronize directly with a phone over Bluetooth/OBEX * accept Bluetooth/OBEX connections in cooperation with obexd 0.19 * run SyncEvolution as a rudimentary HTTP SyncML server These feature were already available in a source-only 1.0 alpha release. For the beta, we fixed some issues (nothing major) and in addition to the source, also make binaries available. As before, we hope to get feedback on where we are going with 1.0 and its SyncML server and direct synchronization features. If you want to get involved, now is a good time because a) there is something which works and b) there is still time to influence the final 1.0, scheduled for March 2010. Documentation of the new features can be found in the "Development" section (http://syncevolution.org/development) for HOWTOs or ask on the mailing list (http://syncevolution.org/support). Here is a more complete list of features compared to the stable release. The full (and up-to-date) list can be retrieved from the Moblin Bugzilla (MB) issue tracking system with this query: http://bugzilla.moblin.org/showdependencytree.cgi?id=7892&hide_resolved=0 For changes compared to the 1.0 alpha please consult the change log. Implemented features are marked with a plus +, open ones with a minus -. ZYB.com + now works thanks to a workaround for anchor handling (MB #2424) - only contacts tested because everything is considered legacy by ZYB.com Slow sync handling (MB #2416) + Unexpected slow syncs can be detected when running as client and if configured (see "preventSlowSync"), abort the session so that the situation can be analyzed. A refresh from client or server might be more suitable. Because this required manual intervention by the user, the feature is off by default. - Catching slow syncs does not work yet when running as server and in one corner case in a client. Improved sync-UI: + settings for HTTP servers are now done inside the list of all configs and server templates instead of poping up a separate window + uses the new D-Bus API + no longer uses private gconf key to select default peer, replaced by "defaultPeer" in SyncEvolution config + added recovery features like handling of unexpected slow syncs (MB #2416) - restoring from backup only supported by command line (MB #8050) - spinner to indicate network activity missing (MB #2229) - interactive password request not implemented yet (MB #6376) Command line: + fixed printing of rejected items (MB #7755) + improved error reporting (textual descriptions instead of plain error codes MB #2069, partial success MB #7755, record and show first ERROR encountered MB #7708) + can create new sources (MB #8424) Redesigned and reimplemented D-Bus API, required by sync-UI: + central syncevo-dbus-server controls configurations and sync sessions: http://syncevolution.org/development/direct-synchronization-aka-syncml-server + accepts incoming SyncML connection requests and messages received by independent transport stubs (obexd, HTTP server, ...) + can be used by multiple user interfaces at once + fully documented, see src/dbus/interfaces + no longer depends on dbus-glib with hand-written glue code for C++, instead uses gdbus plus automatic C++ binding generated via C++ templates - 'syncevolution' command line tool bypasses D-Bus server and runs sync sessions itself (MB #5043) - availability of peers not detected (connected for HTTP, paired for Bluetooth; MB #7700) - Bluetooth peers can only be configured via command line (MB #9216) Revised configuration layout (MB #8048, design document at http://syncevolution.org/development/configuration-handling): + several peer-independent sync and source properties are shared between multiple peers + they can be accessed without selecting a specific peer, by using an empty config name or with the new "@" syntax + user interface in command line and D-Bus API unchanged + old configurations can be read and written, without causing unwanted slow syncs when moving between stable and unstable SyncEvolution versions + old configurations can be migrated with the "--migrate" command line switch; however, then older SyncEvolution can no longer access them and migrating more than one old configuration causes the second or later configuration to loose its "deviceId" property (which is shared now), causing a slow sync once + config names may contain characters that are not allowed in the file names used for the underlying files; will be replaced with underscores automatically (MB #8350) - users of the sync-ui will not know about the --migrate option, so if they have only one configuration, it should be migrated automatically SyncML server in general: + incoming connections are accepted by syncevo-dbus-server via the D-Bus Connection API; because this is a "personal SyncML server", all local data is meant to belong to a single user, and only one sync session can be active at any point in time + different users on the same machine can run their own server, as long as they ensure that listening for incoming connections does not conflict with each other (different port in HTTP) + the session of an HTTP client which stops sending messages expires after "RetryDuration" seconds instead of blocking the server forever (MB #7710) - suspend/resume support is untested (MB #2425) - automatic backup of server databases is inefficient (done even when client is not allowed to do a sync; always backs up all data, including sources which are not active; MB #7708) - the progress events and statistics reported for a SyncML client are not generated when running as SyncML server, will require a fair amount of refactoring in the Synthesis engine (MB #7709) - the Synthesis server example config contains workarounds for specific phones, but SyncEvolution does not currently use those; adding new workarounds should be made very simple (MB #7712) HTTP SyncML server: + test/syncevo-http-server.py provides an experimental HTTP server based on Python and Twisted - a configuration must be created for each peer manually, including a remoteDeviceId value that contains the peer's SyncML device ID (MB #7838) OBEX SyncML server ("sync with phones"): + peers are contacted via a builtin transport that uses libopenobex (MB #5188) + Server Alerted Notification (SAN) message triggers syncs; server ID and URI are configurable (MB #7871) - a configuration must be created for each peer manually, including a syncURL that contains the peer's MAC address (MB #7838) - should be integrated into the system's Bluetooth pairing (MB #7089) OBEX SyncML client: + obexd 0.19 contains a plugin which passes SyncML messages to syncevo-dbus-server - parsing of SAN message is rudimentary and depends on an existing local configuration, needs to be refined depending on which SyncML server software it is meant to work with (MB #6175) Automatic sync (MB #6378): - no support for the various server push notification mechanisms - no intelligent detection of local changes - no regular background sync - depends on safe handling of concurrent editing, which is blocked by merging of a new Evolution Data Server API (MB #3479) Upgrading from 0.9.x: * Upgrading and downgrading should work seamlessly when using existing configurations. But this being an alpha, better ensure that you have backups of both your data and your configurations in ~/.config/syncevolution. * The new configuration layout is only used when creating new configurations or explicitly invoking "syncevolution --migrate" (see above). Such configs cannot be used by older SyncEvolution releases. SyncEvolution 0.9.1 -> 0.9.2, 23.01.2010 ======================================== Synthesis SyncML Engine version: see src/synthesis/ChangeLog New Maemo 5/Nokia N900 calendar backend and packages, brought to you by Ove Kaaven. These packages are available via the Maemo extras-devel repository. Bug reports can be submitted both in http://bugs.maemo.org and http://bugzilla.moblin.org. The latter is the tracker that is monitored by the SyncEvolution team, which will also incorporate patches. In general, Ove is the main maintainer of the new backend. New XMLRPC backend, contributed by Franz Knipp/M-otion. It accesses data inside a web service via a SOUP API and thus allows synchronizing it via SyncML. See src/backends/xmlrpc/README for more information. Added templates for Oracle Beehive and Goosync. Both are not currently part of the regular testing. In addition to that, 0.9.2 is an incremental update, with several updated translations and addressing all of the issues reported by users for 0.9.1: - vCard dialects: added "X-GENDER/X-SIP" (used by Maemo) and X-SKYPE (used by Maemo and recent Evolution, MB #8948) - Evolution Address Book: avoid picking CouchDB by default (MB #7877, evolution-couchdb #479110) CouchDB address books are appended at the end of the local database list, otherwise preserving the order of address books. The initial release of evolution-couchdb in Ubuntu 9.10 is unusable because it does not support the REV property. Reordering the entries ensures that the CouchDB address book is not used as the default database by SyncEvolution, as it happened in Ubuntu 9.10. Users can still pick it intentionally via "evolutionsource". - installation: templates now in $(datadir)/syncevolution/templates (MB #7808) This are files used internally, meant to be extended by distributors. Storing them in /etc is no longer supported, but also unlikely to be needed. Added warnings that these files cannot simply be copied into .config because they are not complete configurations. - installation: "make install" populates $(docdir) (MB #7168) Previously README, COPYING, NEWS, and server READMEs were copied into syncevolution.org .tar.gz/.deb/.rpm archives as part of custom make rules and thus missing in other installations. - building: --with-boost had no effect (MB#7856), detect incorrect use of --with-synthesis-src, workaround for lack of --with-docdir in older autoconf, do not unnecessarily depend on CPPUnit header files and GNOME/EDS libs (MB#8338), workaround for libtool bug ("cannot install `syncecal.la' to a directory not ending in ..."), - clarified documentation of properties for file backend (MB#8146) - stderr redirection: detect "error" messages and show them (MB#7655) The "GConf Error: Failed to contact configuration server..." error message was suppressed by the code which catches noise from libraries invoked by SyncEvolution. Now it is printed as ERROR, making it easier to detect why running SyncEvolution inside cron needs additional changes: http://www.estamos.de/blog/2009/05/08/running-syncevolution-as-cron-job/ - importing contacts from SyncML server without full name (MB#5664): Evolution expects the name to be set and shows an empty string if it is missing. Now the name is re-added by appending first, middle and last name. - Evolution calendar: work around 'cannot encode item' problem (MB #7879) Happens when the calendar file contains broken events which reference a timezone that is not defined. Now the event is treated like one in the local timezone. - "http_proxy" env variable is supported regardless which HTTP transport is used (MB#8177). - avoid crashes when libecal sets neither error nor pointer (MB#8005) and when aborting a running sync in the syncevo-dbus-server (MB#8385) - "--status" output: fixed missing total item counts (MB #9097) Upgrading from 0.9.1: * nothing to do, upgrading and downgrading should work seamlessly SyncEvolution 0.9 -> 0.9.1, 26.10.2009 ====================================== Synthesis SyncML Engine version: see src/synthesis/ChangeLog Mobical and Memotoo are now officially supported. Memotoo uses vCard 2.1 with several Evolution specific extensions. It uses iCalendar 2.0, however, without actually supporting the advanced features of it. Times are converted to UTC and meeting information are lost. Mobical uses vCard 2.1 and vCalendar 1.0 as data formats, with the result that many properties used in Evolution are not supported by the server. In particular calendar support is very limited (known issues when events are in time zones different from the one selected locally and on the server, no support for meetings). For details see README.mobical. *** Beware *** that the Mobical SyncML password is *not* the same as the one for their web site. Log into mobical.net, then go to "my accounts >> configure new device >> manual settings" to find the SyncML credentials. It is now possible to compile database backends outside of SyncEvolution, install them and have SyncEvolution use them automatically like any other backend. The backend API has been enhanced considerably. For example, backend developers have access to a modular set of utility classes that can be mixed into a specific implementation. Backends can access the internal Synthesis representation directly and therefore no longer need their own vCard/vCalendar/iCalendar parser. The sqlite demo backend can be enabled and compiled again with --enable-sqlite. It demonstrates how to map directly from the Synthesis field list to some internal format (an SQLite database schema in this case). Other changes: * Resend messages to cope with intermittent loss of network connectivity (Moblin Bugzilla #3427). See the new "ResendDuration" and "ResendDelay" configuration properties for details. * SyncEvolution command line uses the GNOME keyring when the new --keyring option is given. * The logging of added and updated items was enhanced. Events, tasks and memos are logged with a short description instead of just the local ID. The description for contacts was improved. * Receiving photos from Mobical failed because Mobical does not quite follow the vCard 2.1 (Moblin Bugzilla #6668). Sending photos worked, but added a few bytes of garbage at the end of each photo (typically ignored when showing). Parser was made more tolerant by Synthesis and encoder bug was fixed. * Task priorities used by Mobical and Evolution did not match: vCalendar 1.0 uses 1-3, iCalendar 2.0 uses 1-9 (MB #6664). SyncEvolution now translates between the two ranges, with some information getting lost when talking to a peer which only supports the smaller range. * Importing work and home phone numbers from Google into desktop Evolution works better, because SyncEvolution now adds the "VOICE" flag expected by Evolution (MB#6501). * SSL certificate checking with Google is enabled by default and enabled in Moblin, because libsoup in that distro has the necessary fix. Without that fix, all connection attempts fail. The binaries on syncevolution.org are compiled with --disable-ssl-certificate-check, so users who want the additional security must enable it. * .rpms on syncevolution.org no longer specify a dependency on certain Perl features. This depencency was a problem on Mandriva. Unwanted hard dependencies on libecal in syncevolution.org binaries are avoided for real this time (MB#6552). * Some sync-UI enhancements (describe sync services, avoid crash with very long input in some of the text boxes (MB#5219), set application icon, improved some strings). * sync-UI: now disables sources which are not supported when setting up a configuration, like memos on Moblin (MB #6672). Previously the source was enabled, which prevented using using the configuration as-is on the command line. * The sync UI allowed to enable calendar and task synchronization with Google although Google does not support that (MB#5871). In new installations this is prevented by clearing the URI for those data categories. * Trying to remove a non-existent configuration via the command line now raises an error, to catch typos (MB #6673). * Improved checks which logs in the logdir belong to the current server (MB#5215). * Improved sanity checking of integer configuration parameters (MB#6500). * Spelling fix: "aboring" => "aborting" Known issue: * Mobical and Memotoo do not have a description in the GUI yet. * ZYB.com is not supported because of a known anchor handling problem in the server (MB#2424). Upgrading from 0.9: * nothing to do, upgrading and downgrading should work seamlessly SyncEvolution 0.9.1 beta 2 -> 0.9.1, 26.10.2009 =============================================== Synthesis SyncML Engine version: see src/synthesis/ChangeLog Minor changes: * spelling fixes in NEWS file (--source-type => --source-property) * update to zh_CN * improved autotools compilation of libsynthesis SyncEvolution 0.9.1 beta 1 -> 0.9.1 beta 2, 19.10.2009 ====================================================== Synthesis SyncML Engine version: see src/synthesis/ChangeLog Several fixes: * Receiving photos from Mobical failed because Mobical does not quite follow the vCard 2.1 (Moblin Bugzilla #6668). Sending photos worked, but added a few bytes of garbage at the end of each photo (typically ignored when showing). Parser was made more tolerant by Synthesis and encoder bug was fixed. * Task priorities used by Mobical and Evolution did not match: vCalendar 1.0 uses 1-3, iCalendar 2.0 uses 1-9 (MB #6664). SyncEvolution now translates between the two ranges, with some information getting lost when talking to a peer which only supports the smaller range. * The workaround for detecting an endless stream of Alert 222 messages (caused by misbehavior of certain servers when a specific message has to be resent) aborted certain valid (albeit somewhat pathologic) sync sessions. Improved the heuristic so that it still catches the real loop without aborting in that other case. * sync-ui: now disables sources which are not supported when setting up a configuration, like memos on Moblin (MB #6672). Previously the source was enabled, which prevented using using the configuration as-is on the command line. * .rpms on syncevolution.org no longer specify a dependency on certain Perl features. This depencency was a problem on Mandriva. Unwanted hard dependencies on libecal in syncevolution.org binaries are avoided for real this time (MB#6552). * Trying to remove a non-existent configuration via the command line now raises an error, to catch typos (MB #6673). * Message resend options: added sanity checks to catch negative values, clarified that duration is given in seconds, 0s resend interval disables resending (MB #6500). * Spelling fix: "aboring" => "aborting" SyncEvolution 0.9 -> 0.9.1 beta 1, 06.10.2009 ============================================= Synthesis SyncML Engine version: see src/synthesis/ChangeLog Mobical and Memotoo are now officially supported. Memotoo uses vCard 2.1 with several Evolution specific extensions. It uses iCalendar 2.0, however, without actually supporting the advanced features of it. Times are converted to UTC and meeting information are lost. Mobical uses vCard 2.1 and vCalendar 1.0 as data formats, with the result that many properties used in Evolution are not supported by the server. In particular calendar support is very limited (known issues when events are in time zones different from the one selected locally and on the server, no support for meetings). For details see README.mobical. *** Beware *** that the Mobical SyncML password is *not* the same as the one for their web site. Log into mobical.net, then go to "my accounts >> configure new device >> manual settings" to find the SyncML credentials. It is now possible to compile database backends outside of SyncEvolution, install them and have SyncEvolution use them automatically like any other backend. The backend API has been enhanced considerably. For example, backend developers have access to a modular set of utility classes that can be mixed into a specific implementation. Backends can access the internal Synthesis representation directly and therefore no longer need their own vCard/vCalendar/iCalendar parser. The sqlite demo backend can be enabled and compiled again with --enable-sqlite. It demonstrates how to map directly from the Synthesis field list to some internal format (an SQLite database schema in this case). Other changes: * Resend messages to cope with intermittent loss of network connectivity (Moblin Bugzilla #3427). See the new "ResendDuration" and "ResendDelay" configuration properties for details. * The logging of added and updated items was enhanced. Events, tasks and memos are logged with a short description instead of just the local ID. The description for contacts was improved. * The sync UI allowed to enable calendar and task synchronization with Google although Google does not support that (MB#5871). In new installations this is prevented by clearing the URI for those data categories. * Importing work and home phone numbers from Google into desktop Evolution works better, because SyncEvolution now adds the "VOICE" flag expected by Evolution (MB#6501). * SyncEvolution command line uses the GNOME keyring when the new --keyring option is given. * SSL certificate checking with Google is enabled by default and enabled in Moblin, because libsoup in that distro has the necessary fix. Without that fix, all connection attempts fail. The binaries on syncevolution.org are compiled with --disable-ssl-certificate-check, so users who want the additional security must enable it. * syncevolution.org binaries should be compatible with a wider range of Evolution releases again (MB#6552). * Some sync UI enhancements (describe sync services, avoid crash with very long input in some of the text boxes (MB#5219), set application icon, improved some strings). * Improved checks which logs in the logdir belong to the current server (MB#5215). * Improved sanity checking of integer configuration parameters (MB#6500). Known issue: * Mobical and Memotoo do not have a description in the GUI yet. * ZYB.com is not supported because of a known anchor handling problem in the server (MB#2424). SyncEvolution 0.8.1 -> 0.9, 12.08.2009 -------------------------------------- Synthesis SyncML Engine version: see src/synthesis/ChangeLog This is a major new release, with first steps towards further improvements. From this release on, the Synthesis SyncML engine will be the underlying SyncML and data conversion engine. A native GTK GUI is now included. The "sync-ui" program depends on a backend D-Bus service ("synevo-dbus-server") and several auxiliary files. Therefore, it only runs without hacks after installation in /usr (possible with .deb, .rpm and binary .tar.gz archives, and with "sudo make install", after compiling from source). The normal command line tool still works without being installed. In this release, the data handling model was changed from "all items are sent verbatim to the SyncML server" to "parse and convert". The argument for the former approach was that the SyncML server should be the only entity in the system which does data conversion. The previous releases already had to deviate from this approach to accommodate for minor client/server incompatibilities and for vCard 2.1 support, so the new approach just takes it one step further. The main reason for going to full semantic conversion is vCalendar 1.0 support. Support by servers for iCalendar 2.0, the only format supported by 0.8.1, is often still incomplete or even non-existent. By doing the conversion on the client side, SyncEvolution is now able to synchronize events and tasks with a wider variety of servers. It is still true that properties not supported by a server cannot be synchronized to other devices, so using a server with full iCalendar 2.0 support is recommended. But in contrast to 0.8.1, information that can be stored only locally is no longer lost when receiving an incomplete update from the SyncML server, thanks to intelligent merging, provided by the Synthesis engine. This depends on an accurate description of the server's capabilities, which might not be provided by all of them. This still needs to be tested in more detail. Interoperability with servers tested extensively in this release. The following servers are now supported: * ScheduleWorld. There is very complete support for Evolution data. The only known issues are around resuming from an interrupted sync. * Google contact sync. Google follows the vCard 2.1 specification, and thus does not support some of the vCard 3.0 additions, nor some of the common extensions. As a result, several properties are not synchronized (nickname, birthday, spouse/manager, URLs, ...). Only one top-level organization seems to be supported. For details, see README.google. Regarding Google's SyncML support, refresh-from-client and one-way-from-client sync modes are not supported. Deleting contacts moves them out of the main address without deleting them permanently. When adding such a contact again, the server discards the data sent by the client and recreates the contact with the data that it remembered. Because SSL certificate checking for Google works only with libsoup if the platform has a patched libsoup (http://bugzilla.gnome.org/show_bug.cgi?id=589323) or libsoup >= 2.28, certificate checking remains turned off by default for Google. If your platform has a suitable libsoup (like Moblin 2.0), then enable checking with: syncevolution --configure \ --sync-property SSLVerifyServer=true \ --sync-property SSLVerifyHost=true \ google * Funambol, with calendar and task support. Funambol supports iCalendar 2.0 in the current server, so this is enabled in the configuration template. Not all iCalendar 2.0 features are supported by the server, most notably support for meetings (drops attendees), meeting invitations (drops UID), detached recurrences (drops RECURRENCE-ID). See README.funambol for details. Interoperability with the Funambol server was improved by adding support for some vCard extensions (X-MANAGER/ASSISTANT/SPOUSE/ANNIVERSARY, #2418). Lost ACTION property has a work around (#2422). To enable that support in an existing configuration so that it exchanges items in the more suitable iCalendar 2.0 format, use: syncevolution --configure --source-property sync=two-way \ funambol calendar todo syncevolution --configure --source-property type='calendar:text/calendar!' \ funambol calendar syncevolution --configure --source-property type='todo:text/calendar!' \ funambol todo Without the exclamation mark, format auto-negotiation would pick the less capable vCalendar 1.0 format because that is marked as preferred by the server. *** WARNING ***: After switching from a previous release to the current one, or vice versa, do a "syncevolution --sync refresh-from-server" or "--sync refresh-from-client" (depending on which side has the authoritative copy of the data) once, to get client and server into a consistent state. Not doing so can result in applying the same changes to the server multiple times, and thus duplicates. Other changes in detail: * vCalendar 1.0 is now supported. * Both libcurl and libsoup can be selected at compile time as HTTP(S) transport mechanism. * SF #2101015: Expect: 100-continue header results in 417 Error with proxy. Should no longer occur with the HTTP transports in this release. * SF #1874805: Syncing with Funambol results in loosing all-day property. This now works thanks to the Synthesis data conversion rules. * SF #2586600: Synchronisation with mobical.net fails in 0.8.1. Works now, but there are some known issues (Bugzilla #3009) and therefore mobical.net is not officially supported yet. * SF #2542968: Separator for categories should not be escaped. Done correctly by the Synthesis vcard conversion. * bug fix: Evolution notes with only a summary and no description were not sent correctly to the server. Instead of sending the summary, an empty text was sent. * CTRL-C no longer kills SyncEvolution right away. Instead it asks the server to suspend the session. If that takes too long, then pressing CTRL-C twice quickly will abort the sync without waiting for the server (Warning, this may lead to a slow sync in the next session). * WBXML is enabled by default now, except for Funambol (#2415). Using WBXML reduces message sizes and increases parsing performance. * New configuration templates can be added to /etc/default/applications/syncevolution. These templates may contain icons, which are used by the GUI (no icons shipped right now). * Information about previous synchronization sessions is now stored in a machine-readable format and can be accessed using the new --print-sessions options. The output of this information is more complete and more nicely formatted. * --status now shows not only data changes since the last sync, but also item changes (see README for the difference between the two). * The new --restore option allows restoring local data to the state as it was before or after a sync. For this to work, "logdir" must be set (done by default for new configurations). The format of database dumps was changed to implement this feature. Instead of in a flat file, items are now saved as individual files in a directory. To get the previous format back (for example, to import as one .vcf or .ics file manually) concatenate these files. * With –-remove, one can remove configurations. It leaves data files and the local databases untouched. Known issues: * The GUI includes the number of locally deleted items during a refresh-from-server sync in the number of "received changes" (#5185), which is a bit misleading. This is a result of #3314, which introduced changes not "received" from the server. * When a network error occurs and the client never notices that the connection to the server was lost, it will hang forever, waiting for the server's reply (#3427). * The file backend now works only for data formats understood by SyncEvolution and the Synthesis engine. Items are parsed when exchanging them among the backend, engine, and server, in contrast to 0.8.1, where item content was not touched locally (#5046). * The ZYB.com server sends conflicting sync anchors, so most syncs don't work as expected (#2424). SyncEvolution 0.9 beta 3 hotfix -> 0.9 final, 12.08.2009 -------------------------------------------------------- Because SSL certificate checking for Google only works with libsoup if the platform has a patched libsoup (http://bugzilla.gnome.org/show_bug.cgi?id=589323) or libsoup >= 2.28, certificate checking remains turned off by default for Google. If your platform has a suitable libsoup (like Moblin 2.0), then enable checking with: syncevolution --configure \ --sync-property SSLVerifyServer=true \ --sync-property SSLVerifyHost=true \ google Only minor changes: * updated translations * refresh-from-server syncs now report how many items were deleted locally at the start of the sync (Bugzilla #3314). The GUI includes the number of locally deleted items during a refresh-from-server sync in the number of "received changes", which is a bit misleading (#5185). * fixed build issue on Fedora 11/g++ 4.4 (Bugzilla #5061) * some build and test improvements * proper fix for D-Bus error functions (#4919) * improve sync-ui startup time by avoiding an unnecessary copying of the sync config into itself (#5021) * adapted tooltip style (used for SyncML server links) to new Moblin theme, they weren't visible earlier (#5017) * notify zone selector in Moblin 2.0 about sync-ui startup (#4752) * sync-ui: minor layout change for fatal error situation SyncEvolution 0.9 beta 3 -> 0.9 beta 3 hotfix, 23.07.2009 --------------------------------------------------------- Found additional Google limitation: the server drops photos if they exceed a certain size. The limit is somewhere between 40KB (okay) and 80KB (dropped). The last-minute workaround for Google/libsoup/gnutls (using http) didn't work because apparently Google only supports SyncML over https (Bugzilla #4551). Now the default configuration template uses https with all certificate checking disabled. A patch for libsoup was submitted to upstream. Some error messages by the "syncevolution" command line tool were not printed (#4676). The root cause was the intentional interception of stderr to hide the noise printed by various system libraries (#1333). Unfortunately remarks about incorrect command line options were among swallowed messages. No good workaround available short of disabling the redirection with SYNCEVOLUTION_DEBUG=1, so let's release an update... Other changes: * updated translations SyncEvolution 0.9 beta 2 -> 0.9 beta 3, 21.07.2009 -------------------------------------------------- Synthesis SyncML Engine version: see src/synthesis/ChangeLog Enabled calendar and task synchronization for myFunambol.com. Not all iCalendar 2.0 features are supported by the server, most notably support for meetings (drops attendees), meeting invitations (drops UID), detached recurrences (drops RECURRENCE-ID). See README.funambol for details. Interoperability with the Funambol server was improved by adding support for some vCard extensions (X-MANAGER/ASSISTANT/SPOUSE/ANNIVERSARY, Bugzilla #2418). Lost ACTION property is worked around (#2422). Synchronization with Google Contacts was enabled and tested. A configuration template for that server is now provided. Google follows the vCard 2.1 specification and thus does not support some of the vCard 3.0 additions, nor some of the common extensions. As a result, several properties are not synchronized (nickname, birthday, spouse/manager, URLs, ...). Only one top-level organization seems to be supported. For details, see README.google. Regarding Google's SyncML support, refresh-from-client and one-way-from-client sync modes are not supported. Deleting contacts moves them out of the main address without deleting them permanently. When adding such a contact again, the server discards the data sent by the client and recreates the contact with the data that it remembered. SSL certificate checking with libsoup (the default transport) is now supported (#2431). However, libsoup/gnutls are very strict about SSL certificate checking and reject version 1 certificates, like the one used by Verisign for Google (#4551). At the moment the only solution is to fall back to plain http in the Google configuration template. CTRL-C no longer kills SyncEvolution right away. Instead it asks the server to suspend the session. If that takes too long, then pressing CTRL-C twice quickly will abort the sync without waiting for the server (warning, this may lead to a slow sync in the next session). WBXML is enabled by default now, except for Funambol (#2415). Using WBXML reduces message sizes and increases parsing performance. It was not enabled initially in the 0.9 releases in order to test this new feature more thoroughly. Old configs don't have an explicit enableWBXML setting and therefore will automatically use the new default. Various bug fixes and improvements: * only show servers in GUI which are tested and supported (Bugzilla #3336) * a single log file is written in .html format (#3474) * added several translations of the GUI * lots of testing improvements, build binary packages again UPGRADING When enabling calendar and todo synchronization with Funambol in an existing configuration, set the type so that iCalendar 2.0 is used: syncevolution --configure --source-property sync=two-way funambol calendar todo syncevolution --configure --source-property type='calendar:text/calendar!' funambol calendar syncevolution --configure --source-property type='todo:text/calendar!' funambol todo When creating a configuration anew, this is not necessary because the configuration template contains those types. SyncEvolution 0.9 beta 1 -> 0.9 beta 2, 12.06.2009 -------------------------------------------------- Synthesis SyncML Engine version: see src/synthesis/ChangeLog Major new feature: a GTK GUI! The "sync-ui" program depends on a backend D-Bus service ("synevo-dbus-server") and several auxiliary files. Therefore it only runs without hacks after "sudo make install", in contrast to the normal command line which can be invoked directly. New configuration templates can be added to /etc/default/applications/syncevolution. These templates may contain icons which are used by the GUI (no icons shipped right now). Information about previous synchronization sessions is now stored in a machine-readable format and can be accessed via the new --print-sessions options. The output of this information is more complete and nicer formatted. --status now not only shows data changes since the last sync, but also the item changes (see README for the difference between the two). The new --restore option allows restoring local data to the state as it was before or after a sync. For this to work, "logdir" must be set (done by default for new configurations). The format of database dumps was changed to implement this feature: instead of in a flat file, items are now saved as individual files in a directory. To get the previous format back (for example, to import as one .vcf or .ics file manually) concatenate these files. With --remove one can remove configurations. It leaves data files and the local databases untouched. Various bug fixes and improvements: * compiles and works again on Debian Etch if Boost 1.35 is installed from www.backports.org (without GUI, see Bugzilla #3358) * uses XDG_CACHE_HOME (= ~/.cache) for logs and database dumps to avoid interfering with .desktop search in XDG_DATA_HOME; the directory there is automatically moved when running syncevolution (Bugzilla #3309) * re-enabled certain config options (clientAuthType, maxMsgSize, maxObjSize); normally it shouldn't be necessary to modify those (Bugzilla #3242, #2784) * fixed error handling of unexpected server reply in libsoup transport (Bugzilla #3041) * message logging is enabled at logLevel 3 (XML translation) and 4 (also original XML or WBXML message) * GTK GUI fixes since initial Moblin 2.0 beta: only start it once if libunique is available (Bugzilla #3154), wrap text in change sync service" button (Bugzilla #2064), sort sources alphabetically in UI (Bugzilla #2070) SyncEvolution 0.8.1 -> 0.9 beta 1, 13.05.2009 --------------------------------------------- Synthesis SyncML Engine version: see src/synthesis/ChangeLog A major new release and the first step towards further improvements: from this release onwards, the Synthesis SyncML engine is used as the underlying SyncML and data conversion engine. The focus of this first beta was to reach the same level of functionality and stability as in 0.8.1. Therefore this beta does not yet bring much new features; this will be the focus of further beta releases until finally 0.9 will be a full replacement for 0.8.1. This release also switches from an "all items are sent verbatim to the SyncML server" to a "parse and convert" data handling model. The argument for the former approach was that the SyncML server should be the only entity in the system which does data conversion. The previous releases already had to deviate from this approach to accommodate for minor client/server incompatibilities and for vCard 2.1 support. The main reason for going to full semantic conversion is vCalendar 1.0 support. Support by servers for iCalendar 2.0, the only format supported by 0.8.1, is often still incomplete or even non-existent. By doing the conversion on the client side, SyncEvolution is now able to synchronize events and tasks with a wider variety of servers. It is still true that properties not supported by a server cannot be synchronized to other devices, so using a server with full iCalendar 2.0 support is recommended. But in contrast to 0.8.1, information that can only be stored locally is no longer lost when receiving an incomplete update from the SyncML server thanks to intelligent merging provided by the Synthesis engine. This depends on an accurate description of the server's capabilities, which might not be provided by all of them - still needs to be tested in more detail. *** WARNING ***: after switching from a previous release to the current one or vice versa, do a "syncevolution --sync refresh-from-server" or "--sync refresh-from-client" (depending on which side has the authoritative copy of the data) once to get client and server into a consistent state. Not doing so can result in applying the same changes to the server multiple times and thus duplicates. Changes in detail: * vCalendar 1.0 is now supported. Because this hasn't been tested that much yet, events and tasks are still disabled in the Funambol default configuration (SF #2635973). * Both libcurl and libsoup can be selected at compile time as HTTP(S) transport mechanism. * SF #2101015: Expect: 100-continue header results in 417 Error with proxy Should no longer occur with the HTTP transports in this release. * SF #1874805: Syncing with Funambol results in loosing all-day property This now works thanks to the Synthesis data conversion rules. * SF #2586600: Synchronisation with mobical.net fails Should work now because of the different SyncML implementation (untested). * SF #2542968: separator for categories should not be escaped Done correctly by the Synthesis vcard conversion. * bug fix: Evolution notes with only a summary and no description were not sent correctly to the server: an empty text was sent instead of sending the summary Known shortcomings in this release which will be fixed before the final 0.9: * Verbatim file backups of items on the SyncML server are currently not possible: the SyncEvolution "file" backend still exists, but all items are converted by the Synthesis engine and therefore must be in a format supported by the engine. * HTTPS can be used with libsoup, but certificate checking is always disabled. Need to find a portable way to determine where the certificate file is on various systems. * Log file handling is not yet unified: the traditional client.log contains only high-level SyncEvolution log entries. Low-level SyncML and engine log entries are in sysync_*.html files. * stdout and stderr messages from system libraries are visible on the console. 0.8.1 used to redirect those into the client.log to hide this noise; this will be added again. In the meantime, ignore messages like "Deadlock potential - avoiding evil bug!". This is liborbit telling us that it is (hopefully successfully) handling something nasty. SyncEvolution 0.8.1 -> 0.8.1a, 15.12.2008 ----------------------------------------- C++ client library: 7.0 plus some patches, see github repository referenced in configure script. A minor bug fix release, updating only necessary on Mac OS X. * #2307976 "Trace/BPT trap - sync failure": occurs randomly in Mac OS X specific transport layer of the Funambol C++ client library. Avoided in 0.8.1a by using libcurl as transport, as in 0.7. SyncEvolution 0.8 -> 0.8.1, 11.10.2008 -------------------------------------- C++ client library: 7.0 plus some patches, see github repository referenced in configure script. A minor bug fix release, updating not really necessary. The binary packages for Evolution are built now so that one package works for all compatible Evolution releases, including the new Evolution 2.24. * Evolution calendar: regression in 0.8: one-way sync of virtual birthday calendar (#2095433). "refresh-from-client" works again for the birthday calendar. Other modes are not supported. In contrast to previous releases SyncEvolution now does some sanity checks that the sync mode is right. * Mac OS X: removing old logdirs failed (#2087389). Fixed. * SyncML client library: "Expect: 100-continue" header resulted in 417 error with certain proxies (#2101015). Now this header is always disabled; it doesn't make much sense with SyncML anyway. * The development of the Funambol C++ client library is now tracked in a git repository on github.com. Modifications and tags for SyncEvolution are checked in there. The configure script checks out the right sources from there automatically; can be controlled via --with-funambol-src parameter. * Evolution desktop: the version of the used Evolution libraries is included in the "--version" output and log files. * Cleaned up README. Kudos to Martin Wetterstedt for pointing out mistakes in the README and the web site. SyncEvolution 0.7 -> 0.8, 29.08.2008 ------------------------------------ C++ client library: 7.0 plus compatibility patch for Synthesis Updating user configuration: this version introduces a new, simplified configuration layout. Old configurations still work. They can be converted to the new format via a new "--migrate" command line option. *** WARNING ***: this version uses a different change tracking for Mac OS X address book, Evolution calendars, task lists and memos. After switching from a previous release to the current one or vice versa, do a "syncevolution --sync refresh-from-server" once to reset the change tracking. Not doing so can result in applying the same changes to the server multiple times and thus duplicates. * New configuration file layout: following the freedesktop.org recommendation, new configurations are stored in $XDG_CONFIG_HOME/syncevolution or $HOME/.config/syncevolution if XDG_CONFIG_HOME is not set. The old layout under $HOME/.sync4j/evolution is still supported. * New command line options: new configurations can be created by syncevolution itself (--configure), including setting of all configuration properties (--sync-property, --source-property). The configuration can dumped to stdout (--print-config), with or without comments explaining each property (--quiet). See the README for details. * The "evolutionsource" source property no longer has to be configured. If left blank, the default client database will be synchronized. * Selecting which kind of data is to be synchronized under a specific source name is a lot easier now and the same on all supported platforms: the SyncEvolution backends can be selected via aliases (e.g. "contacts") and the format is specified via an optional MIME type (e.g. "contacts:text/x-vcard"). In the unlikely situation that multiple backends are active which can synchronize the same kind of data, then the right one can be selected by the unique name of the backend (e.g. "Evolution Address Book"). * New configurations automatically get a random client ID string. Setting it manually is still possible, but no longer necessary. Disabling unavailable data sources is also done automatically. SyncEvolution checks that the backend is available and there is at least one database (the first one will be synchronized unless explicitly changed). If these checks fail and the sync source was explicitly requested by the user by listing it after the server name, then an error is printed and no configuration is written. If the user wants the default setup, then the source is silently disabled. * All passwords can be read from stdin at runtime or an environment variable (see "--sync-property password=?" or README for details). Both avoids the less secure storing of plain text passwords in the configuration files (SF #1832458). * Detached recurrences: meeting series where some occurrences were modified are now supported. Previously only the main event was synchronized. All exceptions got lost when copying back from the server. Requires a SyncML server which supports this. ScheduleWorld was extended to do that. * Fixed segfaults caused by logging certain data. The reason was an API change in the client library's logging calls which the older SyncEvolution code hadn't been adapted to. Did not normally occur, but might have been the reason for SF #1830149 (unconfirmed). * Time zone support: the time zones of incoming events are mapped to native time zone definitions whenever possible. Currently this works if the TZID follows the Olson naming scheme with a location at the end. Matching the time zone has the advantage of being able to update the time zone definition without having to recreate the event. If matching fails and the VTIMEZONE definition differs from one already imported earlier, then SyncEvolution works arounds limitation in Evolution by renaming the time zone. Previously the new event used the old and most likely out-dated time zone definition. ***WARNING***: Evolution itself does not do either of these steps itself yet, thus importing meeting invitations via Evolution still fails in some cases. The code implementing the time zone handling described above was written with inclusion into Evolution itself in mind; a discussion with the Evolution developers about that is in progress. * On Maemo/Nokia Internet Tablets, calendar synchronization now works because the new calendar change tracking no longer depends on some of the backend calls which used to fail (SF #1734977). * Added SSL configuration options: certificate checking can be relaxed or disabled completely (SF #1852647). * Added a new file backend: stores each SyncML item as a separate file in a directory. The directory has to be specified via the database name, using [file://] as format. The file:// prefix is optional, but the directory is only created if it is used. Change tracking is done via the file systems modification time stamp: editing a file treats it as modified and then sends it to the server in the next sync. Removing and adding files also works. The local unique identifier for each item is its name in the directory. New files are created using a running count which initialized based on the initial content of the directory to "highest existing number + 1" and incremented to avoid collisions. Although this sync source itself does not care about the content of each item/file, the server needs to know what each item sent to it contains and what items the source is able to receive. Therefore the "type" property for this source must contain a data format specified, including a version for it. Here are some examples: - type=file:text/vcard:3.0 - type=file:text/plain:1.0 * Code restructuring: it is now possible to add new backends and thus write SyncML clients for other kinds of data without touching any line of code in SyncEvolution itself. All the required interfaces are documented inside SyncEvolution itself. A HTML documentation can be built via the new "make doc" target (requires Doxygen and dot). The SyncEvolution framework itself never depended on GNOME or Evolution, only the Evolution data sources did. If you want support for other ways of storing your data, consider writing a new data source - it is really easy. See EvolutionSyncSource or TrackingSyncSource for details. * Messages are printed to the screen immediately. More readable log file format. * Maemo: the useless ''list: unable to access calendars: failure' error message is avoided. It was triggered by not having memo support in Evolution Data Server. Cleaned up the code so that it properly distinguishes between 'calendar', 'memo list' and 'task list'. * added server template for MemoToo; note that the server has not been tested * added synchronization of Evolution memo summary Most devices only synchronize plain text and do not have a separate summary field. Such an extra summary field was added to Evolution after memo support was initially implemented in SyncEvolution, therefore SyncEvolution did not transmit that field. Added transmitting the summary by inserting it as first line of the plain text blob *if* it is not already identical with the first line. When receiving a memo, the summary is set from the first line *without* removing the first line because the first line might have been used as a normal part of the memo. * Various other minor changes, fixes and lots of code cleanups. * license cleanup: SyncEvolution is GPL v2 or later SyncEvolution 0.8 beta 2 -> 0.8 final, 29.08.2008 ------------------------------------------------- C++ client library: 7.0 plus compatibility patch for Synthesis * license cleanup: SyncEvolution is GPL v2 or later SyncEvolution 0.8 beta 2 -> 0.8 beta 3, 17.08.2008 -------------------------------------------------- C++ client library: 7.0 plus compatibility patch for Synthesis * Another revision of updating events in Evolution calendars: the method introduced in 0.8 beta 1 for dealing with detached recurrences did not work with the Evolution Exchange Connector. Now both Exchange and local calendars pass the unit tests again. * minor code cleanup (testing, writing additional backends) SyncEvolution 0.8 beta 1 -> 0,8 beta 2, 03.08.2008 -------------------------------------------------- C++ client library: 7.0 plus compatibility patch for Synthesis * To prevent accidental sync runs when a configuration change was intented, a new --run switch must be used when configuration properties are given on the command line. When neither --run nor --configure are specified, SyncEvolution prints an error and refuses to do anything. * Improved documentation for command line, in particular the synopsis. * Added a new file backend: stores each SyncML item as a separate file in a directory. The directory has to be specified via the database name, using [file://] as format. The file:// prefix is optional, but the directory is only created if it is used. Change tracking is done via the file systems modification time stamp: editing a file treats it as modified and then sends it to the server in the next sync. Removing and adding files also works. The local unique identifier for each item is its name in the directory. New files are created using a running count which initialized based on the initial content of the directory to "highest existing number + 1" and incremented to avoid collisions. Although this sync source itself does not care about the content of each item/file, the server needs to know what each item sent to it contains and what items the source is able to receive. Therefore the "type" property for this source must contain a data format specified, including a version for it. Here are some examples: - type=file:text/vcard:3.0 - type=file:text/plain:1.0 * Code restructuring: it is now possible to add new backends and thus write SyncML clients for other kinds of data without touching any line of code in SyncEvolution itself. All the required interfaces are documented inside SyncEvolution itself. A HTML documentation can be built via the new "make doc" target (requires Doxygen and dot). SyncEvolution 0.8 alpha 1 -> 0.8 beta 1, 12.07.2008 --------------------------------------------------- C++ client library: the frozen 7.0 code, but before the release * Added support for detached recurrences (aka modified instances of a recurring event). Requires a SyncML server which supports this. ScheduleWorld was extended to do that. * Fixed segfaults caused by logging certain data. The reason was an API change in the client library's logging calls which the older SyncEvolution code hadn't been adapted to. Did not normally occur, but might have been the reason for SF #1830149 (unconfirmed). * when creating a config for the first time, only enable sync sources which can be synchronized (SF #1991286) The check for that was completely missing. Now SyncEvolution checks that the backend is available and there is at least one database (the first one will be synchronized unless explicitly changed). If these checks fail and the sync source was explicitly requested by the user by listing it after the server name, then an error is printed and no configuration is written. If the user wants the default setup, then the source is silently disabled. * Fixed incorrect properties in some of the new server templates (ScheduleWorld syncURL + calender URI, Funambol syncURL, ScheduleWorld addressbook type) * Device IDs must start with the "sc-pim-" prefix, otherwise myFUNAMBOL may treat different devices as the single phone that myFUNAMBOL supports, leading to unwanted slow syncs. * Maemo package is build again so that backends are loaded dynamically: installing Dates application is as it was with the 0.7 release (SF #1993109). The useless ''list: unable to access calendars: failure' error message is avoided. It was triggered by not having memo support in Evolution Data Server. Cleaned up the code so that it properly distinguishes between 'calendar', 'memo list' and 'task list'. * added server template for MemoToo; note that the server has not been tested * added synchronization of Evolution memo summary Most devices only synchronize plain text and do not have a separate summary field. Such an extra summary field was added to Evolution after memo support was initially implemented in SyncEvolution, therefore SyncEvolution did not transmit that field. Added transmitting the summary by inserting it as first line of the plain text blob *if* it is not already identical with the first line. When receiving a memo, the summary is set from the first line *without* removing the first line because the first line might have been used as a normal part of the memo. * removed --properties option: it wasn't implemented yet and won't be in 0.8 * fixed regression in alpha 1: setting sync mode during status query or sync affected *all* sources, even the disabled ones. Now it only affects the enabled ones, as intended. To enable disabled sync sources, list them after the server name. *** WARNING ***: this version uses a different change tracking for for Mac OS X AddressBook. After switching from a previous release to the current one or vice versa, do a "syncevolution --sync refresh-from-server" once to reset the change tracking. Not doing so can result in applying the same changes to the server multiple times and thus duplicates. A similar change was necessary in 0.8 alpha 1 for Evolution calendar, tasks, and memos. When switching from a version >= 0.8 alpha 1 to an older version or vice versa also refresh the local databases. 0.8 alpha 1 did not create correct configurations. When you want to continue using such a configuration, make sure that in addition to the obviously wrong syncURLs also the less obvious ScheduleWorld config mistakes are fixed: * calendar: uri=cal2 * addressbook: type=addressbook:text/vcard * deviceId must start with "sc-pim-" if you synchronize with myFUNAMBOL, otherwise there may be unwanted slow syncs when multiple devices with a different deviceId connect. Note that changing the deviceId causes a slow sync, so you should get client and server in sync before changing the value, change it, then do a "--sync refresh-from-server". SyncEvolution 0.7 -> 0.8 alpha 1, 19.04.2008 -------------------------------------------- C++ client library: a snapshot of the development version Updating user configuration: this version introduces a new, simplified configuration layout. Old configurations still work. They can be converted to the new format via a new "--migrate" command line option. *** WARNING ***: this version uses a different change tracking for Evolution calendars, task lists and memos. After switching from a previous release to the current one or vice versa, do a "syncevolution --sync refresh-from-server" once to reset the change tracking. Not doing so can result in applying the same changes to the server multiple times and thus duplicates. * New configuration file layout: following the freedesktop.org recommendation, new configurations are stored in $XDG_CONFIG_HOME/syncevolution or $HOME/.config/syncevolution if XDG_CONFIG_HOME is not set. The old layout under $HOME/.sync4j/evolution is still supported. * New command line options: new configurations can be created by syncevolution itself (--configure), including setting of all configuration properties (--sync-property, --source-property). The configuration can dumped to stdout (--print-config), with or without comments explaining each property (--quiet). See the README for details. * The "evolutionsource" source property no longer has to be configured. If left blank, the default client database will be synchronized. * Selecting which kind of data is to be synchronized under a specific source name is a lot easier now and the same on all supported platforms: the SyncEvolution backends can be selected via aliases (e.g. "contacts") and the format is specified via an optional MIME type (e.g. "contacts:text/x-vcard"). In the unlikely situation that multiple backends are active which can synchronize the same kind of data, then the right one can be selected by the unique name of the backend (e.g. "Evolution Address Book"). * New configurations automatically get a random client ID string. Setting it manually is still possible, but no longer necessary. * All passwords can be read from stdin at runtime or an environment variable (see "--sync-property password=?" or README for details). Both avoids the less secure storing of plain text passwords in the configuration files (SF #1832458). * Detached recurrences: meeting series where some occurrences were modified are now supported. Previously only the main event was synchronized. All exceptions got lost when copying back from the server. ***WARNING***: such events are accepted by ScheduleWorld, but not propagated to other clients. Under investigation. * Time zone support: the time zones of incoming events are mapped to native time zone definitions whenever possible. Currently this works if the TZID follows the Olson naming scheme with a location at the end. Matching the time zone has the advantage of being able to update the time zone definition without having to recreate the event. If matching fails and the VTIMEZONE definition differs from one already imported earlier, then SyncEvolution works arounds limitation in Evolution by renaming the time zone. Previously the new event used the old and most likely out-dated time zone definition. ***WARNING***: Evolution itself does not do either of these steps itself yet, thus importing meeting invitations via Evolution still fails in some cases. The code implementing the time zone handling described above was written with inclusion into Evolution itself in mind; a discussion with the Evolution developers about that is in progress. * On Maemo/Nokia Internet Tablets, calendar synchronization now works because the new calendar change tracking no longer depends on some of the backend calls which used to fail (SF #1734977). * Added SSL configuration options: certificate checking can be relaxed or disabled completely (SF #1852647). * Adding support for new local data sources is easier now. The SyncEvolution frame work itself never depended on GNOME or Evolution, only the Evolution data sources did. If you want support for other ways of storing your data, consider writing a new data source - it is really easy. See EvolutionSyncSource or TrackingSyncSource for details. * Messages are printed to the screen immediately. More readable log file format. * Various other minor changes and fixes. SyncEvolution 0.6 -> 0.7, 17.12.2007 ------------------------------------ C++ client library: r_6_5_3_1 Updating user configuration: no relevant changes in this release. For those who haven't done so already, enabling large object support is recommended (see syncml/config.txt sample configs). * added port for iPhone and Mac OS X Address Book * fixed Nokia packaging problem which prevented installation via the package manager unless it was in "red pill" mode (SF #1781652) * sync with eGroupware - lost or messed up telephones: SyncEvolution incorrectly added TYPE=OTHER to phone numbers sent with e.g. CELL instead of TYPE=CELL (SF #1796086). Another patch was required for eGroupware itself to correctly map phone numbers as sent by SyncEvolution, see Compatibility web page. * added .deb packages * adapted calendar event insert/update to Evolution 2.12: the UID needs to be restored, otherwise the Evolution backend crashes (GNOME issue #488881) * new feature: if the previous log directory is still available, then local changes made since last sync can be queried before starting a sync (new option --status) and will be printed directly before a sync. Setting the "logdir" option will automatically keep the most recent logs and database dumps around. * added command line options: --sync|-s Temporarily synchronize the active sources in that mode. Useful for a 'refresh-from-server' or 'refresh-from-client' sync which clears all data at one end and copies all items from the other. --status|-t The changes made to local data since the last synchronization are shown without starting a new one. This can be used to see in advance whether the local data needs to be synchronized with the server. --quiet|-q Suppresses most of the normal output during a synchronization. The log file still contains all the information. --help|-h Prints usage information. --version Prints the SyncEvolution version. * default configurations now reference the normal Evolution databases ("Personal") thus requiring less changes to use. The account information is now clearly marked as placeholder which needs to be entered. * bugfix: vCard 3.0 with mixed case were not converted properly to vCard 2.1 by SyncEvolution (must convert to upper case because vCard 2.1 only allows that), leading to problems with mapping phone numbers in the Funambol server. Diagnosed and reported by Paul McDermott, thanks a lot! * support receiving plain text notes with \n and \r\n line breaks; always send with \r\n * added explicit error message when syncevolution is invoked with incorrect names in the list of sources to synchronize: previously it silently ignored unknown names * improved output: less verbose ("extracting" items is now logged at debug level and thus not normally shown) and more informative printing of changes (table summarizes number of changes on client and server, heading for comparison changed to make it clear that it shows changes on the client) * SyncCap is not generated unless syncModes are configured: added a comment to example config (SF #1764123) * improved error handling: catch errors during post-processing and continue SyncEvolution 0.7-pre2 -> 0.7, 17.12.2007 ----------------------------------------- C++ client library: r_6_5_3_1 * bugfix: vCard 3.0 with mixed case were not converted properly to vCard 2.1 by SyncEvolution (must convert to upper case because vCard 2.1 only allows that), leading to problems with mapping phone numbers in the Funambol server. Diagnosed and reported by Paul McDermott, thanks a lot! * support receiving plain text notes with \n and \r\n line breaks; always send with \r\n * added explicit error message when syncevolution is invoked with incorrect names in the list of sources to synchronize: previously it silently ignored unknown names * added stack dumping in case of premature abort; removed workaround for lost connection to Evolution Dataserver again because the workaround itself caused random segfaults inside glib SyncEvolution 0.7-pre1 -> 0.7-pre2, 08.11.2007 ---------------------------------------------- C++ client library: branch b_v65 Updating user configuration: no relevant changes in this release. For those who haven't done so already, enabling large object support is recommended (see syncml/config.txt sample configs). It is required for myFUNAMBOL to synchronize very large address books and some users have reported segfaults unless this option was enabled. * iPhone bug fix: syncing contacts with photos was unreliable (export) and crashed (import) because the API had not been called correctly * iPhone + ScheduleWorld: when configured to use vcard3 (recommended!) then contacts are exchanged as vCard 3.0 * iPhone + ScheduleWorld bugfix: importing vCard 3.0 did not correctly classify the phone numbers. A sync with the new "--sync refresh-from-server" option will fix this, assuming that the server has the correct data. * Evolution: detect a crashed backend and abort SyncEvolution instead of hanging forever. * adapted calendar event insert/update to Evolution 2.12: the UID needs to be restored, otherwise the Evolution backend crashes (GNOME issue #488881) * new feature: if the previous log directory is still available, then local changes made since last sync can be queried before starting a sync (new option --status) and will be printed directly before a sync. Setting the "logdir" option will automatically keep the most recent logs and database dumps around. * added command line options: --sync|-s Temporarily synchronize the active sources in that mode. Useful for a 'refresh-from-server' or 'refresh-from-client' sync which clears all data at one end and copies all items from the other. --status|-t The changes made to local data since the last synchronization are shown without starting a new one. This can be used to see in advance whether the local data needs to be synchronized with the server. --quiet|-q Suppresses most of the normal output during a synchronization. The log file still contains all the information. --help|-h Prints usage information. --version Prints the SyncEvolution version. * default configurations now reference the normal Evolution databases ("Personal") thus requiring less changes to use. The account information is now clearly marked as placeholder which needs to be entered. SyncEvolution 0.6 -> 0.7-pre1, 17.10.2007 ----------------------------------------- * C++ client library: tag "sdkcpp_6_0_9_1" (same as before) * added support for Mac OS X/iPhone address book * fixed Nokia packaging problem which prevented installation via the package manager unless it was in "red pill" mode * improved output: less verbose ("extracting" items is now logged at debug level and thus not normally shown) and more informative printing of changes (table summarizes number of changes on client and server, heading for comparison changed to make it clear that it shows changes on the client) * example configs were in share/share directory (SF #1767329) * Nokia 770/800: uninstallable package fixed by setting category (SF #1781652) * sync with eGroupware - lost or messed up telephones: SyncEvolution incorrectly added TYPE=OTHER to phone numbers sent with e.g. CELL instead of TYPE=CELL (SF #1796086). Another patch was required for eGroupware itself to correctly map phone numbers as sent by SyncEvolution, see Compatibility web page. * SyncCap is not generated unless syncModes are configured: added a comment to example config (SF #1764123) * improved error handling: catch errors during post-processing and continue SyncEvolution 0.5 -> 0.6, 13.07.2007 ------------------------------------ * C++ client library: tag "sdkcpp_6_0_9_1" * added support for synchronizing Evolution notes (aka memos) as plain text where the first line serves as summary; this is the format understood by ScheduleWorld * added support for synchronizing Evolution notes (aka memos) as iCal 2.0 journal; not currently supported by any server and untested * revamped example configs and documentation: only one set of config files for each server is provided, because this is more likely to be needed by users * example configs are now installed in share/doc/syncevolution, enabled message limit and large object support in them * added support for Nokia 770/800 (aka Maemo): built with loadable modules so that it works with whatever backends are installed, improved log handling to accomodate for limited space on filesystem (see below), some workarounds * added workaround for Nokia 770: contacts are not really deleted unless the EDS-Sync with instant messaging servers is activated; now SyncEvolution will delete contacts marked as deleted by the GUI before a sync if it finds any. WARNING: if you use EDS-Sync and SyncEvolution, then give EDS-Sync enough time after going online to finish its own synchronization of modified/deleted contacts before starting SyncEvolution. * improved log handling: writing log and database dumps can be disabled with "logdir=none", verbosity of log is controlled by "loglevel", better handling of errors during initial database access * added workaround for Evolution bug #455274: the separator for multiple categories in events and tasks is not generated and interpreted according to iCalendar 2.0 by Evolution; as a consequence of that items sent to the server had all categories merged into one and items imported into Evolution only used one of the catories http://bugzilla.gnome.org/show_bug.cgi?id=455274 * fixed off-by-one counting of months in backup directory names * fixed error handling: a failed source was not forced into a slow sync as required; one failed source prevented saving configs of not-failed ones and thus forced those into an unnecessary slow sync * uses the Funambol C++ testing framework (which is based on the previous SyncEvolution testing); now creates its configs and (when using CLIENT_TEST_EVOLUTION_PREFIX=file://) also the Evolution databases automatically * implemented synccompare as pure Perl script using Algorithm::Diff instead of external diff tool * synccompare did not figure out width of shell window as it should have * better error handling if creating the before/after database dumps fails (SF #1685637) * workaround for Funambol 3.0 trailing = parser bug UPGRADING Old config files from 0.5 or older continue to work, but it is recommended to set the following options to enable message size limits: maxMsgSize = 8192 maxObjSize = 500000 loSupport = 1 SyncEvolution 0.6pre2 -> 0.6, 13.07.2007 ---------------------------------------- * improved README/HACKING documents * fixed the new example configs: use event/task for Funambol 6.0, name was wrong * added workaround for Evolution bug #455274: the separator for multiple categories in events and tasks is not generated and interpreted according to iCalendar 2.0 by Evolution; as a consequence of that items sent to the server had all categories merged into one and items imported into Evolution only used one of the catories http://bugzilla.gnome.org/show_bug.cgi?id=455274 * added workaround for Nokia 770: contacts are not really deleted unless the EDS-Sync with instant messaging servers is activated; now SyncEvolution will delete contacts marked as deleted by the GUI before a sync if it finds any. WARNING: if you use EDS-Sync and SyncEvolution, then give EDS-Sync enough time after going online to finish its own synchronization of modified/deleted contacts before starting SyncEvolution. SyncEvolution 0.6pre1 -> 0.6pre2, 23.04.2006 -------------------------------------------- * C++ client library: tag "sdkcpp_6_0_7" + revision 1.7 of build/autotools/test/Makefile.am * added support for synchronizing Evolution notes (aka memos) as plain text where the first line serves as summary; this is the format understood by ScheduleWorld, not the iCal 2.0 format added in 0.6pre1 * improved log handling: writing log and database dumps can be disabled with "logdir=none", verbosity of log is controled by "loglevel", better handling of errors during initial database access * fixed off-by-one counting of months in backup directory names * fixed error handling: a failed source was not forced into a slow sync as required; one failed source prevented saving configs of not-failed ones and thus forced those into an unnecessary slow sync * revamped example configs: only one set of config files for each server is provided, because this is more likely to be needed by users * uses the Funambol C++ testing framework (which is based on the previous SyncEvolution testing); now creates its configs and (when using CLIENT_TEST_EVOLUTION_PREFIX=file://) also the Evolution databases automatically SyncEvolution 0.5 -> 0.6pre1, 26.03.2006 ---------------------------------------- * C++ client library: CVS snapshot from 26.03.2006 * added support for synchronizing Evolution notes (aka memos) as iCal 2.0 journal * added --enable-static-cxa = linking C++ runtime statically: binaries produced for 0.6 will have less external dependencies than the 0.5 binaries * added hacks for Maemo/Nokia 770, including a build mode with dynamically loadable modules (--enable-shared, --enable-maemo, --with-patched-dbus) * implemented synccompare as pure Perl script using Algorithm::Diff instead of external diff tool * synccompare did not figure out width of shell window as it should have * better error handling if creating the before/after database dumps fails (SF #1685637) * example configs are now installed in share/doc/syncevolution, enabled message limit and large object support in them * workaround for Funambol 3.0 trailing = parser bug UPGRADING Old config files continue to work, but it is recommended to set the following options to enable message size limits: maxMsgSize = 8192 maxObjSize = 500000 loSupport = 1 SyncEvolution 0.4 -> 0.5, 12.11.2006 ------------------------------------ * C++ client library revision "syncevolution-0-5": - added support for sending changes in smaller chunks ("Large Object Support"): disabled by default, see updated example configuration - time is printed with GMT offset so that a server admin in a different timezone can always figure out how a client log relates to events on the server - special item keys as they might be stored in some calendars after importing non-Evolution events are now properly supported * bug fix: in 0.4 it was necessary to manually configure the verDTD or the Funambol 3.0a server would choke on the invalid SyncML during the second synchronization with SyncEvolution; now this option is set automatically * added support and testing of transmitting just the changes from client to server or vice versa; see "one-way-from-server/client" in example configuration * fixed/updated comments in the example configuration * improved automated testing and fixed the problem that CPPUnit was not found unless it was part of the system * Now works on Maemo/Nokia 770: minor changes were necessary so that the system address book can now be selected under the name "<>. Copying 300 contacts into the Nokia 770 went fine, but any further attempt to synchronize suffered from timeouts inside the embedded Evolution Data Server. SyncEvolution 0.3 -> 0.4, 11.09.2006 ------------------------------------ * C++ client library revision "syncevolution-0-4": - added support for device information, required by some servers - fixed incompatibilities with non-Funambol servers - the user agent string can now be modified in the spds/syncml/config.txt, but it is recommended to not set it explicitly. Then SyncEvolution will automatically insert its current version. - #305795: for tasks the "text/x-todo" type from the configuration was sent to servers instead of the correct "text/calendar" provided by SyncEvolution itself - sync modes "refresh-client/server" can now be specified as "refresh-from-client/server" in the config * updated default syncml/config.txt: - firstTimeSyncMode has never been implemented in the library, removed its documentation, - added documentation for userAgent - use "refresh-from-client/server" * SF issue 1511951: support copying changes back from EGroupware server by not expecting the UID of calendar items to be unmodified * fixed a bug where after a refresh-from-client sync changes would be sent to the server again during a two-way sync although the server already had them * implemented authentication for Evolution databases * synccompare was removing too many parts of vCards with single-value ORG properties * improved error reporting when selected server is not configured * changed vCard parser to make it compatible with servers which send a verbatim semicolon as part of properties where the semicolon has no special meaning * If minor errors occur like not being able to insert an item at the client or server side, then it is reported in the log and output, but the next synchronization will be a normal synchronization, not a forced slow one as in previous versions. The old approach ensured that the problem was noticed and fixed, but required user assistance. With the new approach synchronization continues to work, although without fixing the root cause of the problem. * Workaround for bug in Evolution 2.0.6 (and perhaps other versions): for calendars and task lists not all deleted items were reported at once thus a single synchronization would only tell the server about a subset of the changes. Repeating the synchronization would eventually be told of all changes, so now this repetition is built into the code which queries for changes and a single synchronization is sufficient as it should be. SyncEvolution 0.4 pre2 -> 0.4, 11.09.2006 ----------------------------------------- * adapted to C++ client library from CVS head, tagged as syncevolution-0-4: devinfo.patch patch was merged with several changes to the API * SF issue 1511951: support copying changes back from EGroupware server by not expecting the UID of calendar items to be unmodified SyncEvolution 0.4 pre1 -> pre2, 21.08.2006 ------------------------------------------ * C++ client library revision "syncevolution-0-4-pre2": most patches were merged into CVS head, but .patches/devinfo.patch still needs to be applied manually when checking out from the Funambol CVS instead of using the bundled version * fixed a bug where after a refresh-from-client sync changes would be sent to the server again during a two-way sync although the server already had them * implemented authentication for Evolution databases * synccompare was removing too many parts of vCards with single-value ORG properties * improved error reporting when selected server is not configured * use 7-bit quoted-printable encoding with explicit UTF-8 charset for vCard 2.1 to avoid any potential confusion about the content; not really necessary because SyncML specifies 8-bit UTF-8 as the default * fix for 0.4 pre 1: sending CHARSET is not allowed (and not needed) for vCard 3.0, so it was removed again (did not harm either) * fix for 0.4 pre 1: sending vCard 2.1 to Synthesis server did not work because the new device info always mentioned 3.0 as the preferred format - now the preferred format matches the one that was configured and that thus will be used. SyncEvolution 0.3 -> 0.4 pre 1, 2006-08-06 ------------------------------------------ * C++ client library revision "funambol30ga" plus the patches stored in its ".patches" directory: - the user agent string can now be modified in the spds/syncml/config.txt, but it is recommended to not set it explicitly. Then SyncEvolution will automatically insert its current version. - now compatible with additional servers (fixed some SyncML protocol issues, added support for sending device information) - revised API of the client library - #305795: for tasks the "text/x-todo" type from the configuration was sent to servers instead of the correct "text/calendar" provided by SyncEvolution itself - sync modes "refresh-client/server" can now be specified as "refresh-from-client/server" in the config * updated default syncml/config.txt: - firstTimeSyncMode has never been implemented in the library, removed its documentation, - added documentation for userAgent - use "refresh-from-client/server" * changed vCard parser to make it compatible with servers which send a verbatim semicolon as part of properties where the semicolon has no special meaning * If minor errors occur like not being able to insert an item at the client or server side, then it is reported in the log and output, but the next synchronization will be a normal synchronization, not a forced slow one as in previous versions. The old approach ensured that the problem was noticed and fixed, but required user assistance. With the new approach synchronization continues to work, although without fixing the root cause of the problem. * Workaround for bug in Evolution 2.0.6 (and perhaps other versions): for calendars and task lists not all deleted items were reported at once thus a single synchronization would only tell the server about a subset of the changes. Repeating the synchronization would eventually be told of all changes, so now this repetition is built into the code which queries for changes and a single synchronization is sufficient as it should be. * Made it compile on Maemo 2.0, the Nokia 770 build environment, by adding "--disable-ecal". Not tested yet, though. SyncEvolution 0.3, 2006-06-27 ----------------------------- * added syncing of calendars and tasks as iCalendar 2.0 * added syncing of contacts as vCard 3.0 * tested extensively with sync.scheduleworld.com and added an example configuration for it * uses C++ client library revision "wmplugin_3_0_20" which contains several bug fixes, among them proper support for special characters and memory handling fixes * much nicer listing of changes made during a sync, handled by the improved "synccompare" utility script (formerly known as "normalize_vcard") * improved automated testing SyncEvolution 0.2, 2006-03-19 ----------------------------- * added automatic backup mechanism and log storage, see "Automatic Backups and Logging". * output no longer is the original log data, but rather a human-readable report of errors and synchronization results. * "normalize_vcard" can now also compare two .vcf files directly. * improved unit tests to catch more errors * hide certain differences in vcards coming back from the server: duplication of extended vcard properties, missing TYPE=OTHER * fixed client library problems: see http://forge.objectweb.org/tracker/?group_id=96&atid=100096 #304792, #304829 * added some more problems to the "Known Problems" section SyncEvolution 0.1, 2006-03-13 ----------------------------- * initial release