Commit Graph

107 Commits

Author SHA1 Message Date
Patrick Ohly 50148ab580 scripting: prevent premature loop timeouts
The more complex "avoid data loss during merging" scripting ran for longer
than 5s limit under extreme conditions (full logging, busy system, running
under valgrind), which resulted in aborting the script and a 10500 "local
internal error" sync failure.

The endless loop prevention should only be necessary to detect programming
mistakes, so better disable it entirely.
2014-10-10 03:19:27 -07:00
Patrick Ohly 2e2c9dca77 vcard: fix caching of PBAP contacts (FDO #84710)
After changing PBAP to send raw items, caching them led to unnecessary
disk writes and bogus "contacts changed" reports. That's because
the merge script relied on the exact order of properties, which was
only the same when doing the redundant decode/encode on the PBAP side.

Instead of reverting back to sending re-encoded items, better enhance
the contact merge script such that it detects contacts as unchanged
when just the order of entries in the property arrays is different.
This relies on an enhanced libsynthesis with the new RELAXEDCOMPARE()
and modified MERGEFIELDS().
2014-10-09 21:31:31 +02:00
Patrick Ohly 936ccc5df2 vcard: remove duplicate loops
Due to cut-and-paste, some arrays were checked multiple times. Only
caused unnecessary overhead.
2014-10-09 21:26:02 +02:00
Patrick Ohly 626e027d3d datatypes: raw text items with minimal conversion (FDO #52791)
When using "raw/text/calendar" or "raw/text/vcard" as SyncEvolution
"databaseFormat", all parsing and conversion is skipped. The backend's
data is identical to the item data in the engine. Finding duplicates
in a slow sync is very limited when using these types because the entire
item data must match exactly.

This is useful for the file backend when the goal is to store an exact copy
of what a peer has or for limited, read-only backends (PBAP).

The implementation uses the "raw" base type which does nothing but storing
the item content verbatim. That type requires a field named "ITEMDATA". After
renaming the variable used by our MAKETEXTWITHPROFILE/PARSETEXTWITHPROFILE
with other, more complex types, all we have to do is skip this conversion
by not providing a profile name.
In a refresh-from-remote local sync of 1000 contacts with photo data
using the file backend, this reduces the number of CPU cycles from
2861476184 to 1840952543 on the server side.
2014-09-12 11:38:40 +02:00
Patrick Ohly adf2522ad1 datatypes: text/calendar+plain revised heuristic, II
When sending a VEVENT, DESCRIPTION was set to the SUMMARY if empty. This may
have been necessary for some peers, but for notes (= VJOURNAL) we don't know
that (hasn't been used in the past) and don't want to alter the item
unnecessarily, so skip that part and allow empty DESCRIPTION.
2014-09-08 11:07:31 +02:00
Patrick Ohly 3392cbf267 datatypes: text/calendar+plain revised heuristic
When receiving a plain text note, the "text/calendar+plain" type
used to store the first line as summary and the rest as description.
This may be correct in some cases and wrong in others.

The EDS backend implemented a different heuristic: there the first
line is copied into the summary and stays in the description. This
makes a bit more sense (the description alone is always enough to
understand the note). Therefore and to avoid behavioral changes
for EDS users when switching the EDS backend to use text/calendar+plain,
the engine now uses the same approach.

For this to work in all cases, the datatype must always call
MEMO_INCOMING_SCRIPT.
2014-07-31 20:20:34 +02:00
Patrick Ohly 5d3df97cac datatypes: fix contact caching
Adding grouping to the contact datatype broke PBAP caching: when
sending an empty URL, for example, during the sync, the parsed contact
had different field arrays than the locally stored contact, because the
latter was saved without the empty URL.

This caused the field-based comparison to detect a difference even when
the final, reencoded contact wasn't different at all.

To solve this, syncing now uses the same "don't send empty properties"
configuration as local storages. Testing shows that this resolves
the difference for EDS.

A more resilient solution would be to add a check based on the encoded
data, but that's more costly performance wise.
2014-07-14 13:38:55 +02:00
Patrick Ohly 7d66823e5f datatypes: fix vCard handling
The new "preserve repeating properties during conflict resolution"
feature was only active when using EDS as storage. The relevant
merge script must be applied to all datatypes, not just the EDS
flavor.

The feature was also unintentionally active when running in
caching mode. This caused two problems:
- The cached item was updated even though only the
  ordering of repeating properties had been modified during
  merging.
- The merged item was sent back to the client side, which
  was undesirable (caching is supposed to be one-way) or even
  impossible (PBAP is read-only, causing sync failures eith error 20030).

We must check for caching mode and disable merging when it is active.
We also must not tell the engine that we updated the photo property
in the winning item, because then that item would get sent to the
read-only side of the sync.

Perhaps a better solution would be to actually tell the engine
that the remote side is read-only when we activate caching mode.
2014-07-11 11:57:14 +02:00
Patrick Ohly 1431fd4a7b datatypes: avoid PHOTO corruption during merge (FDO #77065)
When handling an update/update conflict (both sides of the sync have an
updated contact) and photo data was moved into a local file by EDS, the engine
merged the file path and the photo data together and thus corrupted the photo.

The engine does not know about the special role of the photo property.
This needs to be handled by the merge script, and that script did not
cover this particular situation. Now the loosing side is cleared,
causing the engine to then copy the winning side over into the loosing
one.

Found by Renato Filho/Canonical when testing SyncEvolution for Ubuntu 14.04.
2014-07-11 11:57:14 +02:00
Patrick Ohly 9c0916bee8 vcard profile: avoid data loss during merging
When resolving a merge conflict, repeating properties were taken
wholesale from the winning side (for example, all email addresses). If
a new email address had been added on the loosing side, it got lost.

Arguably it is better to preserve as much data as possible during a
conflict. SyncEvolution now does that in a merge script by checking
which properties in the loosing side do not exist in the winning side
and copying those entries.

Typically only the main value (email address, phone number) is checked
and not the additional meta data (like the type). Otherwise minor
differences (for example, both sides have same email address, but with
different types) would lead to duplicates.

Only addresses are treated differently: for them all attributes
(street, country, city, etc.) are compared, because there is no single
main value.

When copying properties which may have labels, the new entries must be
added such that a unique label can be copied or set. Other properties
can be added at the end of their array.
2014-05-19 21:33:45 +02:00
Patrick Ohly 8c12816dec vcard profile: avoid X-ABLabel without a real property value
The modern profile typically doesn't generate empty properties, and therefore
it is useless to create an IMPP, X-ABRELATEDNAMES or X-ABDATE entry with has
an empty value. It just has the effect of creating X-ABLabels which are not
attached to any property.
2014-05-19 21:33:45 +02:00
Patrick Ohly 933d10f97c CardDAV: use Apple/Google/CardDAV vCard flavor
In principle, CardDAV servers support arbitrary vCard 3.0
data. Extensions can be different and need to be preserved. However,
when multiple different clients or the server's Web UI interpret the
vCards, they need to agree on the semantic of these vCard extensions.

In practice, CardDAV was pushed by Apple and Apple clients are
probably the most common clients of CardDAV services. When the Google
Contacts Web UI creates or edits a contact, Google CardDAV will
send that data using the vCard flavor used by Apple.

Therefore it makes sense to exchange contacts with *all* CardDAV
servers using that format. This format could be made configurable in
SyncEvolution on a case-by-case basis; at the moment, it is
hard-coded.

During syncing, SyncEvolution takes care to translate between the
vCard flavor used internally (based on Evolution) and the CardDAV
vCard flavor. This mapping includes:

X-AIM/JABBER/... <-> IMPP + X-SERVICE-TYPE

  Any IMPP property declared as X-SERVICE-TYPE=AIM will get
  mapped to X-AIM. Same for others. Some IMPP service types
  have no known X- property extension; they are stored in
  EDS as IMPP. X- property extensions without a known X-SERVICE-TYPE
  (for example, GaduGadu and Groupwise) are stored with
  X-SERVICE-TYPE values chosen by SyncEvolution so that
  Google CardDAV preserves them (GroupWise with mixed case
  got translated by Google into Groupwise, so the latter is used).

  Google always sends an X-ABLabel:Other for IMPP. This is ignored
  because the service type overrides it.

  The value itself also gets transformed during the mapping. IMPP uses
  an URI as value, with a chat protocol (like "aim" or "xmpp") and
  some protocol specific identifier. For each X- extension the
  protocol is determined by the property name and the value is the
  protocol specific identifier without URL encoding.

X-SPOUSE/MANAGER/ASSISTANT <-> X-ABRELATEDNAMES + X-ABLabel

  The mapping is based on the X-ABLabel property attached to
  the X-ABRELATEDNAMES property. This depends on the English
  words "Spouse", "Manager", "Assistant" that Google CardDAV
  and Apple devices seem to use regardless of the configured
  language.

  As with IMPP, only the subset of related names which have
  a corresponding X- property extension get mapped. The rest
  is stored in EDS using the X-ABRELATEDNAMES property.

X-ANNIVERSARY <-> X-ABDATE

  Same here, with X-ABLabel:Anniversary as the special case
  which gets mapped.

X-ABLabel parameter <-> property

  CardDAV vCards have labels attached to arbitrary other properties
  (TEL, ADR, X-ABDATE, X-ABRELATEDNAMES, ...) via vCard group tags:
  item1.X-ABDATE:2010-01-01
  item1.X-ABLabel:Anniversary

  The advantage is that property values can contain arbitrary
  characters, including line breaks and double quotation marks,
  which is not possible in property parameters.

  Neither EDS nor KDE (judging from the lack of responses on the
  KDE-PIM mailing list) support custom labels. SyncEvolution could
  have used grouping as it is done in CardDAV, but grouping is not
  used much (not at all?) by the UIs working with the vCards in EDS
  and KDE. It seemed easier to use a new X-ABLabel parameter.

  Characters which cannot be stored in a parameter get converted
  (double space to single space, line break to space, etc.) during
  syncing. In practice, these characters don't appear in X-ABLabel
  properties anyway because neither Apple nor Google UIs allow entering
  them for custom labels.

  The "Other" label is used by Google even in case where it adds no
  information. For example, all XMPP properties have an associated
  X-ABLabel=Other although the Web UI does not provide a means to edit
  or show such a label. Editing the text before the value in the UI
  changes the X-SERVICE-TYPE parameter value, not the X-ABLabel as for
  other fields.

  Therefore the "Other" label is ignored by removing it during syncing.

X-EVOLUTION-UI-SLOT (the parameter used in Evolution to determine the
order of properties in the UI) gets stored in CardDAV. The only exception
is Google CardDAV which got confused when an IMPP property had both
X-SERVICE-TYPE and X-EVOLUTION-UI-SLOT parameters set. For Google,
X-EVOLUTION-UI-SLOT is only sent on other properties and thus ordering
of chat information can get lost when syncing with Google.

CardDAV needs to use test data with the new CardDAV vCard flavor.
Most CardDAV servers can store EDS vCards and thus
Client::Source::*::testImport passed (with minor tweaks in
synccompare) when using the default eds_contact.vcf, but
Client::Sync::*::testItems fails when comparing synced data with test
cases when the synced data uses the native format and the test cases
are still the ones from EDS.

A libsynthesis with URLENCODE/DECODE() and sharedfield parameter for
<property> is needed.
2014-05-19 21:33:38 +02:00
Patrick Ohly 74b0da36b7 vcard profile: remove unused fields
The fields and their offsets were originally used for Synthesis vCard
extensions (X-CustomLabel-* and X-Synthesis-Ref). SyncEvolution never
used those. The extra fields cause unecessary overhead (logging,
merging, memory even when empty), so better remove them.
2014-05-16 11:26:55 +02:00
Patrick Ohly 4b2b2bb64e engine: UID support in contact data
Before, the UID property in a vCard was ignored by the engine.
Backends were responsible for ensuring that the property is
set if required by the underlying storage.

This change moves this into the engine:
- UID is now field. It does not get used for matching
  because the engine cannot rely on it being stored
  by both sides.
- It gets parsed if present, but only generated if
  explicitly enabled (because that is the traditional
  behavior).
- It is never shown in the DevInf's CtCap
  because the Synthesis engine would always show it
  regardless whether a rule enabled the property.
  That's because rules normally only get triggered
  after exchanging DevInf and thus DevInf has to
  be rule-independent. We don't want it shown because
  then merging the incoming item during a local sync
  would use the incoming UID, even if it is empty.
- Before writing, ensure that UID is set.

When updating an existing item, the Synthesis engine reads
the existing item, preserves the existing UID unless the peer
claims to support UID, and then updates with the existing UID.

This works for local sync (where SyncEvolution never claims
to support UID when talking to the other side). It will break
with peers which have UID in their CtCap although they
rewrite the UID and backends whose underlying storage cannot
handle UID changes during an update (for example, CardDAV).

A backend must still activate the UID property by setting m_backendRule
to LOCALSTORAGE-WITH-UID instead of the default LOCALSTORAGE.
Custom backend rules must include the new sub-rule HAVE-VCARD-UID.

See "[os-libsynthesis] UID + CardDAV" for a discussion of this change.
Long term it would be better to configure profiles without depending
on the rule hack.
2014-05-02 16:43:53 +02:00
Patrick Ohly 3347c585ff engine: clean up contacts profile
The "show" attribute of the property values is not supported
and was silently ignored by the Synthesis engine and thus had
no effect. Remove the useless attributes.
2014-05-02 16:43:53 +02:00
Patrick Ohly a496f9dbfc Akonadi: ensure that UID is set (FDO #74342)
Akonadi resources do not enforce iCalendar 2.0 semantic like
"each VEVENT must have a UID" (see "[Kde-pim] iCalendar semantic").
When receiving an event from a peer which itself does not enforce
that semantic (Funambol, vCalendar 1.0 based phones), then we
need to generate a UID, otherwise KOrganizer will ignore the
imported event.
2014-03-26 09:48:52 +01:00
Patrick Ohly ddc1e53b0c PBAP: incremental sync (FDO #59551)
Depending on the SYNCEVOLUTION_PBAP_SYNC env variable, syncing reads
all properties as configured ("all"), excludes photos ("text") or
first text, then all ("incremental").

When excluding photos, only known properties get requested. This
avoids issues with phones which reject the request when enabling
properties via the bit flags. This also helps with
"databaseFormat=^PHOTO".

When excluding photos, the vcard merge script as used by EDS ensures
that existing photo data is preserved. This only works during a slow
sync (merge script not called otherwise, okay for PBAP because it
always syncs in slow sync) and EDS (other backends do not use the
merge script, okay at the moment because PIM Manager is hard-coded to
use EDS).

The PBAP backend must be aware of the PBAP sync mode and request a
second cycle, which again must be a slow sync. This only works because
the sync engine is aware of the special mode and sets a new session
variable "keepPhotoData". It would be better to have the PBAP backend
send CTCap with PHOTO marked as not supported for text-only syncs and
enabled when sending PHOTO data, but that is considerably harder to
implement (CTCap cannot be adjusted at runtime).

beginSync() may only ask for a slow sync when not already called
for one. That's what the command line tool does when accessing
items. It fails when getting the 508 status.

The original goal of overlapping syncing with download has not been
achieved yet. It turned out that all item IDs get requested before
syncing starts, which thus depends on downloading all items in the current
implementation. Can be fixed by making up IDs based on the number of
existing items (see GetSize() in PBAP) and then downloading later when
the data is needed.
2013-07-10 13:08:02 +02:00
Patrick Ohly 6d3b1cf64b EDS: update PHOTO+GEO during slow sync, avoid rewriting PHOTO file
If PHOTO and/or GEO were the only modified properties during a slow
sync, the updated item was not written into local storage because
they were marked as compare="never" = "not relevant".

For PHOTO this was intentional in the sample config, with the
rationale that local storages often don't store the data exactly as
requested. When that happens, comparing the data would lead to
unnecessary writes. But EDS and probably all other local SyncEvolution
storages (KDE, file) store the photo exactly as requested, so not
considering changes had the undesirable effect of not always writing
new photo data.

For GEO, ignoring it was accidental.

A special merge script handles EDS file:// photo URIs. When the
loosing item has the data in a file and the winning item has binary
data, the data in the file may still be up-to-date, so before allowing
MERGEFIELDS() to overwrite the file reference with binary data and
thus forcing EDS to write a new file, check the content. If it
matches, use the file reference in both items.
2013-07-10 13:08:02 +02:00
Patrick Ohly dfd0bc29af EDS contacts: avoid unnecessary DB writes during slow sync due to FILE-AS
EDS 3.8 sets X-EVOLUTION-FILE-AS to "last, first" when a contact is
created without it. This leads again to unnecessary DB updates because
the incoming item that the engine works with doesn't have that field
set.

To mitigate that issue, set FILE_AS (renamed to make the field name
valid in a script) like EDS would do during writing.

The downside is that all incoming items now have FILE_AS set, which
will overwrite a locally stored copy of that property even if the peer
did not store X-EVOLUTION-FILE-AS. Previously, as tested by
testExtension, the local value was preserved. There is no good solution
that works for both use cases, so allow X-EVOLUTION-FILE-AS to get lost
and relax the test.
2013-07-05 17:44:10 +02:00
Patrick Ohly d2a43b1e50 EDS contacts: avoid unnecessary DB writes during slow sync
Traditionally, contacts were modified shortly before writing into EDS
to match with Evolution expectations (must have N, only one CELL TEL,
VOICE flag must be set). During a slow sync, the engine compare the
modified contacts with the unmodified, incoming one. This led to
mismatches and/or merge operations which end up not changing anything
in the DB because the only difference would be removed again before
writing.

To avoid this, define datatypes which include the EDS modifications in
its <incomingscript>. Then the engine works with an item as it would
be written to EDS and correctly determines that the incoming and DB
items are identical.

Found in testpim.py's testSync when applied to the test cases
generated by the Openismus test case generator.
2013-07-05 17:44:10 +02:00
Patrick Ohly 0b35ac1e6d engine: enable out-of-order command execution for SyncEvolution<->SyncEvolution
To take full advantage of batch operations, we must enable out-of-order
execution of commands. Otherwise the engine will never issue more than
one change to us. Do it for SyncEvolution as peer.

TODO: do it for all peers or disable batched processing when it is not
enabled.
2013-07-05 17:44:09 +02:00
Patrick Ohly 762d6c0910 Nokia: always add TYPE=INTERNET to EMAIL (FDO #61784)
Without the explicit TYPE=INTERNET, email addresses sent to a Nokia
e51 were not shown by the phone and even got lost eventually (when
syncing back?).

This commit ensures that the type is set for all emails sent to any
Nokia phone, because there may be other phones which need it and
phones which don't, shouldn't mind. This was spot-checked with a N97
mini, which works fine with and without the INTERNET type.

This behavior can be disabled again for specific Nokia phones by
adding a remote rule which sets the addInternetEmail session variable
to FALSE again.

Non-Nokia phones can enable the feature in a similar way, by setting
the variable to TRUE.
2013-03-05 16:12:48 +01:00
Patrick Ohly b81aa5ab3c engine: support GEO property (part of FDO #60374)
Was lost earlier during syncing. Must be defined in field list and
vCard profile. Still not supported by PIM Manager, because folks doesn't
support it (see FDO #60373).
2013-02-26 12:03:42 +01:00
Patrick Ohly e8adfadcee vCalendar 1.0 + tasks: DUE date could be shifted by a day (FDO #55238)
Because of incomplete support for time conversion, the due date
could get mixed up when phone and PC were set to something other
than UTC.

Reported and fixed by Peter Jan.
2012-10-05 10:22:23 +00:00
Patrick Ohly 49608e282e Funambol: work around PHOTO TYPE=image/jpeg, II
The final version of the fix hadn't made it into the source code.

RFIND() needs a non-zero offset, otherwise it'll not find the slash.
2012-09-13 13:59:08 +02:00
Patrick Ohly b7736253f3 Funambol: work around PHOTO TYPE=image/jpeg
A combination of Funambol Android and Funambol server recently led to
the Funambol server sending PHOTO data with TYPE=image/jpeg. This is
invalid and caused EDS to reject the photo (Vladimir Elisseev,
"[SyncEvolution] issues with syncing photos").

Work around the problem by only keeping the part of the type after the
last slash, if there is any. For image/jpeg and similar types that
leads to the desired value and does not affect valid values, because
those do not contain a slash
(http://www.iana.org/assignments/media-types/image/index.html).
2012-08-14 09:27:49 +02:00
Patrick Ohly 8b86963466 Funambol: ignore UID
Funambol's OneMedia sends UID, but not RECURRENCE-ID. That becomes a
problem when multiple events of the same event series are added to a
backend which follows the iCalendar 2.0 standard (CalDAV, EDS, KDE),
because these events all look like the master event, and there can be
only one of those.

SyncEvolution now strips the UID from all events coming from any
Funambol server (regardless of the version). If a future Funambol
server release adds support for both UID and RECURRENCE-ID, then
SyncEvolution will have to be updated to take advantage of the
improved server. Because the RECURRENCE-ID is also getting
stripped (despite not being set at the moment), SyncEvolution should
continue to work as it does now even if the server changes.

It would have been nice to limit this workaround to affected Funambol
server versions, but an inquiry on the Funambol mailing list didn't
get a reply, therefore SyncEvolution is playing it safe and assumes
that all future Funambol releases will have the same problem.
2012-06-29 16:56:13 +02:00
Patrick Ohly 266656d8bd engine: add DTSTAMP+LAST-MODIFIED before writing calendar items
When writing calendar items into a backend storage as iCalendar 2.0 or
vCalendar 1.0, they should have DTSTAMP and LAST-MODIFIED values. DTSTAMP
is expected by some CalDAV servers (like Apple). LAST-MODIFIED is usually
added by the storage, but not always.

In the text/plain -> syncevolution -> text/calendar -> Radicale -> EDS
-> syncevolution chain the LAST-MODIFIED was not added by Radicale, which caused
problems for change tracking in an EDS-based SyncEvolution.
2012-06-29 11:36:32 +02:00
Patrick Ohly 7ce9b7ff25 WebDAV: improved VJOURNAL -> plain/text conversion
When a memo in VJOURNAL format from the CalDAV backend gets converted
to plain text *and* the description starts with summary plus newline,
then only the description is used in the plain text version, to avoid
duplicating the summary.

Such entries are produced by SyncEvolution when importing plain text
into EDS. The conversion in the CalDAV backend uses a slightly
different logic and strips the first line from the incoming text
instead of just copying it.

The main reason for this change is that the new mechanism (based
libsynthesis text profile) makes it easier to implement the conversion
that way.
2012-06-19 15:45:48 +02:00
Patrick Ohly 92d7e7cc48 engine: support VJOURNAL and VJOURNAL+text/plain
The "text/calendar" type now also contains rules for parsing and
generating VJOURNAL in iCalendar 2.0 and vCalendar 1.0. When using
"text/calendar+plain", the internal format is iCalendar 2.0 with plain
text used as alternative if the sync peer does not support VJOURNAL
for memos.

These types can be used as "databaseFormat" of the file backend and by
other backends which want to replace plain text as exchange format for
memos.
2012-06-15 14:20:25 +02:00
Patrick Ohly 314f073638 local + remote sync: negotiate UID support via SyncCap (BMC #22783)
This uses the new libsynthesis support for adding and checking entries
in the SyncCap to detect per datastore whether UID/RECURRENCE-ID are
truly globally unique and thus can be used to finding pairs. The
presence of the property alone is no guarantee for that.

Previously this kind of pairing was enabled only for local sync, which
was a hack which didn't work for local backends which didn't support
UID (for example, Maemo 5 calendar). It also didn't work for mixtures
of datastores with and without that kind of support.

"1122583000" was randomly chosen as pseudo sync mode. It is a number
because strings confuse Funambol. Note that SYNCMODESUPPORTED() only
works inside the compare script.
2012-05-03 09:45:36 +02:00
Patrick Ohly 1ffa1a66fd sync: fixed inlining of EDS 3.4 PHOTO data
EDS 3.4 introduces PHOTO;VALUE=uri properties where the actual
file path contains a %2F and thus is encoded once more as %252F.
The previous sync script did not support URI decoding and thus
tried to read with the wrong %252F in the file name.

Fixed by relying on a new libsynthesis utility function for decoding
URIs.
2012-03-06 13:38:19 +00:00
Patrick Ohly e9e9a9f34a SyncEvolution<->SyncEvolution sync: allow multiple sync cycles per session
When recognizing that the peer is SyncEvolution, enable the "allow
multiple sync cycles" feature. Does not check whether SyncEvolution is
recent enough! Should better be replaced with a DevInf extension.
2012-03-06 14:03:21 +01:00
Patrick Ohly 4fc6c3e2c4 Merge remote-tracking branch 'origin/syncevolution-1-2-branch'
Conflicts:
	configure.ac
	src/syncevo-dbus-server.cpp
	src/syncevo/Cmdline.cpp
	test/ClientTest.cpp
	test/ClientTest.h
	test/generate-html.xsl
	test/runtests.py

Conflicts caused by backporting + merging back slightly modified
backports.
2011-11-28 10:16:57 +01:00
Patrick Ohly c044c580c2 vCard: X- chat extensions were limited to one instance per kind
For example, only one Jabber account could be synchronized. This
was caused by an incomplete definition of the conversion to and from
vCard.

The test for multiple X- chat extensions is currently only done
in combination with Synthesis because that server is known
to support multiple instances of them.
2011-11-25 13:58:28 +00:00
Patrick Ohly 2a10722652 Maemo Calendar: fix EXDATEs after reading from storage
If the event has a DTSTART with TZID, then the EXDATE also should
have that same TZID. It is uncertain whether the backend provides
the TZID, but even if it does, because of the SIMPLE-EXDATE rule
the value wouldn't be parsed.
2011-11-09 11:38:14 +01:00
Patrick Ohly a3059ee06c Maemo Calendar: avoid TZID in EXDATE property
This must be done for regular EXDATE values in the EXDATE array field
(new SIMPLE-EXDATE rule) and for the addition EXDATE values created
for RECURRENCE-IDs in the EXDATES_DETACHED array field (new
HAVE-EXDATE-DETACHED-NO-TZID rule).

Both these rules are activated as subrules by the new MAEMO-CALENDAR
rule, which is set by the Maemo Calendar backend now.

There is one caveat the SIMPLE-EXDATE rule is also active when parsing
an EXDATE created by the storage and therefore TZID will be ignored,
if any is set at all (uncertain).

A vCalendar outgoing script could fix this by adding the DTSTART time
zone to the floating time value in the parsed EXDATEs.
2011-11-09 11:38:14 +01:00
Patrick Ohly fe1f5b6aac engine: tell peers about detached recurrence exceptions
The Maemo 5 calendar backend does not support UID/RECURRENCE-ID
semantic. This is problematic in combination with peers which rely on
that to override the main recurring event on specific recurrences,
because the Maemo 5 calendar will show duplicates.

Fixing this inside the calendar backend is hard because it doesn't get
to see all related events at once. The engine has the same
problem. Therefore a workaround is used:
- backends which have that unified access to all related events
  add special X-SYNCEVOLUTION-EXDATE-DETACHED properties to the
  main event which correspond to the exceptions represented by
  detached recurrences, in addition to the regular EXDATE; the
  backend must set the HAVE-SYNCEVOLUTION-EXDATE-DETACHED rule when
  using PARSETEXTWITHPROFILE()
- the engine parses the special property (only if the
  HAVE-SYNCEVOLUTION-EXDATE-DETACHED rule is active!) and stores
  the values in the EXDATES_DETACHED field
- this gets passed between SyncEvolution instances in a local
  or SyncML sync
- another SyncEvolution backend can request to get either
  X-SYNCEVOLUTION-EXDATE-DETACHED or regular EXDATE properties
  by setting the HAVE-[SYNCEVOLUTION-]EXDATE-DETACHED rules

The Maemo 5 backend needs to set HAVE-EXDATE-DETACHED. This means that
importing events will add additional EXDATEs which will be sent back
when updating events inside the calendar app, but that is okay. They
are merely redundant.
2011-11-09 11:38:14 +01:00
Patrick Ohly 2df7754eda CardDAV: X-EVOLUTION-UI-SLOT supported by CardDAV peers
It was errouneously concluded that the parameter is not supported.
But it got removed by SyncEvolution, not the servers, because
the extension was not enabled in the profile used for CardDAV.
2011-10-25 15:27:55 +00:00
Patrick Ohly ecfe3b1b46 Evolution: reduced overhead
When the LOCALSTORAGE rule with <noemptyproperties>yes</noemptyproperties>
was introduced, the intention was to make the same change (not storing
empty properties) also in the Evolution backend. Somehow that was never
committed. Doing it now.
2011-10-20 15:20:17 +02:00
Patrick Ohly 4a303e140e vCalendar 1.0: fixed recurring all-day event support
vCalendar 1.0 cannot represent all-day events. The workarounds for
mapping iCalendar 2.0 all-day events into vCalendar 1.0 was
incomplete, leading to effects like shifting EXDATEs and end
times. Example below.

The fix involves several changes:
- EXDATE needs the same conversion="autodate" conversion, so that
  date-only values are sent as date-time when converting to vCalendar 1.0
- End date/time (RR_END) must be converted back to date-only, taking
  the local time zone into account to get correct results.
- EXDATE was converted back to date-only, but without the conversion
  to the local time zone. That shifted the exception dates by one day.
- This last conversion was done for recurring events unless they were
  hourly. In other words, it was done also for non-all-day events.
  This happens to work, although opinions vary whether an all-day EXDATE
  really is valid with a non-all-day DTSTART. The comments in
  bugzilla.moblin.org #3009 where this code was introduced mention a
  bug in Evolution/libical around DATE-TIME EXDATE. Whatever that problem
  was, it no longer occurs in Evolution 2.32 with libical from Debian
  Stable. Time to remove this workaround and only apply the conversion
  when really needed (all-day events).

Example before this fix, from testing against Mobical (aka EverDroid),
after adding new test cases:

Client_Sync_eds_event_testItems.A.test.dat | Client_Sync_eds_event_testItems.B.test.dat
                     only in left file <
                                       > only in right file
-------------------------------------------------------------------------------
BEGIN:VCALENDAR                          BEGIN:VCALENDAR
VERSION:2.0                              VERSION:2.0
  BEGIN:VEVENT                             BEGIN:VEVENT
  SUMMARY:all-day recurrence\, daily\      SUMMARY:all-day recurrence\, daily\
   , with exceptions                        , with exceptions
  DESCRIPTION:recurrs seven times\, e      DESCRIPTION:recurrs seven times\, e
   xcluding (but counting) Friday and       xcluding (but counting) Friday and
   Saturday                                 Saturday
  DTEND;VALUE=DATE:20060407            |   DTEND:20060406T215959Z
  DTSTART;VALUE=DATE:20060406          |   DTSTART:20060405T220000Z
                                       >   EXDATE:20060406
  EXDATE:20060407                          EXDATE:20060407
  EXDATE:20060408                      |   RRULE:FREQ=DAILY;UNTIL=20060412T215
  RRULE:FREQ=DAILY;UNTIL=20060412      |    959Z
  END:VEVENT                               END:VEVENT
END:VCALENDAR                            END:VCALENDAR
-------------------------------------------------------------------------------
BEGIN:VCALENDAR                          BEGIN:VCALENDAR
VERSION:2.0                              VERSION:2.0
  BEGIN:VEVENT                             BEGIN:VEVENT
  SUMMARY:recurrence\, daily\, with e      SUMMARY:recurrence\, daily\, with e
   xceptions                                xceptions
  DESCRIPTION:recurrs seven times\, e      DESCRIPTION:recurrs seven times\, e
   xcluding (but counting) Friday and       xcluding (but counting) Friday and
   Saturday                                 Saturday
  DTEND:20060406T190000Z                   DTEND:20060406T190000Z
  DTSTART:20060406T183000Z                 DTSTART:20060406T183000Z
  EXDATE:20060407T183000Z              |   EXDATE:20060407
  EXDATE:20060408T183000Z              |   EXDATE:20060408
  RRULE:FREQ=DAILY;UNTIL=20060412T183      RRULE:FREQ=DAILY;UNTIL=20060412T183
   000Z                                     000Z
  END:VEVENT                               END:VEVENT
END:VCALENDAR                            END:VCALENDAR
-------------------------------------------------------------------------------
2011-10-20 11:46:35 +02:00
Patrick Ohly a7f7c8eacf Merge remote branch 'origin/syncevolution-1-2-branch'
Conflicts:
	configure.ac
	test/ClientTest.cpp
	test/testcases/eds_event.ics.funambol.tem.patch

Conflicts because of version number and updated test cases resp. local
delete optimization.

ActiveSync backend had to be adapted to modified InsertItemResult: now
it requests a merge when it detects duplicates, like the CalDAV backend
already did on the 1.2 branch.
2011-10-17 13:16:33 +02:00
Patrick Ohly 532c4478d9 iCalendar 2.0: must set VALUE in EXDATE (part of BMC #22940)
EXDATE has a VALUE parameter, which wasn't defined in the XML
profile. Didn't seem to matter at all in practice, but wasn't
standard-compliant. Noticed while working on BMC #22940.
2011-09-14 15:07:31 +02:00
Patrick Ohly 64942372d6 autotools: fixed "make dist[check]" in non-recursive Automake
The bundled libsynthesis source needs to go into "src/synthesis".

None of the XML config files and templates were installed when using
out-of-tree compilation in a clean build directory, because then the
$(wildcard) macro didn't actually match any files. Now it checks the
source directory and the extra directory prefix is removed before
using the resulting file list.
2011-09-02 09:20:23 +00:00
Krzesimir Nowak a7ad98fabc Port build system to non-recursive Automake.
All but toplevel Makefile.am are replaced with their non-recursive
counterparts. The generation of configure.in was removed (and thus
configure-{pre,post}.in are also removed) in favor of configure.ac
and m4 macros adding backend specific configure parts.

Version number is generated like in old build system.

There are still many things to improve, but for now there are no
immediate regressions. AUTOTOOLS-TODO contains a list of possible
improvements and fixes. AUTOTOOLS-TESTING contains what was tested
with current build system (configure flags, make options).
2011-08-30 16:38:34 +02:00
Patrick Ohly c293f51bba CalDAV/CardDAV + local storage: avoid empty properties
The main motivation for this change is that a recent Apple Calendar
server rejects vCards with empty BDAY property. Another reason is that
keeping the data as small as possible is desirable by itself.

Sending an empty property serves as a hint for the peer that the
property is supported. This is not necessary when storing an item in a
backend. Therefore this commit disables empty properties for all
backends which do not themselves set the m_backendRule Synthesis info
value.

The Evolution backend is not affected by this because it has its own
backend rule. That will be updated separately.
2011-08-26 09:45:04 +02:00
Patrick Ohly 6ce041fba8 Sony Ericsson: use ISO-8859-1 for all devices (BMC #14414)
Sony Ericsson seems to use ISO-8859-1 for all their phones. This
causes two problems:
- mangled characters and/or
- crashes inside libecal/gdbus

It is uncertain whether all devices have this problem. A poll did not
yield any results
(http://syncevolution.org/blogs/pohly/2011/question-sony-ericsson-users-charset). So
let's change it for those who have reported problems.

To revert the change, copy
/usr/share/syncevolution/xml/remoterules/server/00_sony_ericsson.xml
into $HOME/.config/syncevolution-xml/remoterules/server (after
creating that directory) and remove the lines with ISO-8859-1.

This was originally reported for contacts, and now also for calendar
data. The calendar case was seen as a crash of the syncevo-dbus-server:

GLib-CRITICAL **: g_variant_new_string: assertion `g_utf8_validate (string, -1, NULL)' failed

Program received signal SIGSEGV, Segmentation fault.
g_variant_is_trusted (value=0x0)
     at /build/buildd/glib2.0-2.28.6/./glib/gvariant-core.c:600
        in /build/buildd/glib2.0-2.28.6/./glib/gvariant-core.c

Thread 1 (Thread 0xb7fce850 (LWP 3402)):
     at /build/buildd/glib2.0-2.28.6/./glib/gvariant-core.c:600
     at /build/buildd/glib2.0-2.28.6/./glib/gvariant.c:3081
     at /build/buildd/glib2.0-2.28.6/./glib/gvariant.c:4093
     app=0xbfffca4c) at /build/buildd/glib2.0-2.28.6/./glib/gvariant.c:4248
     at /build/buildd/glib2.0-2.28.6/./glib/gvariant.c:4188
     in_calobj=0x8435fe8 "BEGIN:VEVENT\r\nSUMMARY:THW Sim Pr\374fung\r\nDTSTART:20070420T230000Z\r\nDTEND:20070421T225900Z\r\nBEGIN:VALARM\r\nTRIGGER;VALUE=DURATION:PT45M\r\nACTION:DISPLAY\r\nEND:VALARM\r\nEND:VEVENT\r\n",
2011-08-10 13:01:47 +02:00
Patrick Ohly 79b65dff05 vcard: adapted PHOTO data inlining to latest libsynthesis
The encoding of PHOTO data which uses plain text when possible is
now only done by libsynthesis if conversion="BLOB_AUTO" is chosen
explicitly.
2011-07-25 15:07:49 +02:00
Patrick Ohly d6d6e8ca39 vcard: inline local photo data (BMC #19661)
Use the new READ() script method in libsynthesis to inline local photo
data right before sending to a remote peer.

Tests were added in combination with SyncEvolution server for inlining
a special well-known file (testcases/local.png) and failing to inline
(file doesn't exist). In the latter case the URI is sent unchanged.
2011-07-22 09:35:03 +02:00
Patrick Ohly eeded54cf4 vCard: support VALUE parameter, fixed TYPE (part of BMC #19661)
The VALUE parameter was ignored completely and thus got lost in the
Synthesis engine. Added it as a string.

The TYPE parameter was an incomplete enum. Better allow arbitrary
strings.

Both parameters must be copied together with the PHOTO data they
belong to. In combination with merge="fillempty" this can be
problematic: if one of the parameters is empty, it may be overwritten
although the PHOTO data is not copied.

This problem is solved by ensuring that the internal field list never
has empty PHOTO_TYPE/VALUE fields. This is done by setting "binary"
resp. "unknown" when importing contact data from a peer or the local
Evolution backend and removing them again before sending to a peer or
storing.

The same change needs to be done with other backends. It is not made
mandatory because some backends (like file) might want to store these
values explicitly.

Because it is unknown which peers support VALUE=uri, only the
CLIENT_TEST_SERVER=syncevolution testItem testcases contain a contact
with such a PHOTO.
2011-07-21 16:09:49 +02:00