Overview
========
The code in this directory only gets compiled when SyncEvolution is
configured with --enable-dbus-service-pim. It uses libfolks and the
PBAP backend to implement a unified address book. This additional
functionality is provided via the org._01.pim D-Bus API.
That API is independent of libfolks and SyncEvolution. That it is
implemented inside syncevo-dbus-server merely simplifies the
implementation, by reusing some code provided by SyncEvolution
and the core Server class:
* C++ D-Bus bindings
* logging
* D-Bus server life cycle control: delay shut down while
clients make calls, inhibit shutdown while clients are
registered, restart when files change on disk during system
update, ...
* direct access to SyncEvolution instead having to go through
the http://api.syncevolution.org D-Bus API
Compilation
===========
Use --enable-dbus-service-pim --enable-pbap --enable-ebook.
--enable-ebook is already the default at the moment. It must not be
disabled.
The PBAP backend can be disabled. However, then fetching data from
phones via PBAP obviously does not work.
More information
================
Public discussion started here, comments can be sent via the gmane
"reply" feature:
http://comments.gmane.org/gmane.comp.mobile.syncevolution/4009
Issues specific to PIM Manager can be found in this graph:
https://bugs.freedesktop.org/showdependencygraph.cgi?id=55916&display=web&rankdir=TB
Dependencies
============
A fairly recent libfolks > 0.7.4 is required. The 837a88 commit
is required and several other pending changes are recommended:
https://bugzilla.gnome.org/show_bug.cgi?id=686693
writing birthday lacks conversion from UTC
https://bugzilla.gnome.org/show_bug.cgi?id=685401
linking by email
https://bugzilla.gnome.org/show_bug.cgi?id=686695
support nickname in add_persona_from_details
Other requirements:
* Evolution Data Server >= 3.6
* boost::locale and libphonenumber (when using the default
sorting and searching)
* glib >= 2.30
* Python >= 2.7 (only for testing)
Known issues
============
None at the moment.
Extending the implementation
============================
Sorting and searching can be replaced with a different implementation
at compile time via --enable-dbus-service-pim[=<locale>]: it uses the
file src/dbus/server/pim/locale-factory-<locale>.cpp to implement
sorting and searching.
Any additional library dependencies for that file need to be added to
the main configure.ac.
The default implementation is based on boost::locale and
libphonenumber and is described below.
Sorting
=======
The sort order can be "first/last", "last/first", "fullname".
"first/last" sorts based on the first name stored in the "name"
property, with the last name used to break ties between equal
first names.
"last/first" reverts that comparison.
"fullname" sorts based on the full name chosen for the contact if
there is such a string, otherwise it uses the concatenation of the
individual name componts without prefix (= "[<first name>]
[<middle name>] [<last name>] [<suffix>])" as fallback. In other
words, it sorts correctly when either all contacts have a full
name explicitly set or the full names that were set following the
same pattern as the fallback.
Sorting is case-insensitive. The default is "last/first" if not set
earlier.
In addition, an empty string as sort order picks a simple, ASCII-based
"last/first" sorting. This is used for testing.
Searching
=========
Supported searches:
[ ] - An empty list matches all contacts.
[ [ <search> ], [ 'limit', <number> ] ] - a 'limit' search
term with a number as parameter (formatted as string) can be added
at the top level term to truncate the search results
after a certain number of contacts. Example: Search([['any-contains',
'Joe'], ['limit', '10']]) => return the first 10 Joes.
As with any other search, the resulting view will be updated if
contact data changes.
The limit must not be changed in a RefineSearch(). A 'limit' term
may (but doesn't have to) be given. If it is given, its value must
match the value set when creating the search. This limitation
simplifies the implementation and its testing. The limitation
could be removed if there is sufficient demand.
[ [ <search> ] ] - the same as [ <search> ]
[ 'or', <search1>, <search2>, ... ] - combines 0 to n other
searches and results in a match if any of the sub-searches
matches. ['or'] without any sub-search does not match.
[ 'and', <search1>, <search2>, ... ] - like 'or', but matches if
and only if all of the sub-searches match.
[ 'phone', '<number>' ] - Look up a valid phone number (= "caller ID").
The country code for the current locale is added if no country
code was given in the number. Phone numbers in the unified address
book must start with the resulting full number, after being normalized
the same way.
In other words:
- Formatting does not matter.
- Alpha characters are aliases for numbers on the keypad and match
their corresponding number.
- Additional digits in the address book are ignored, only
the prefix must match (extensions may or may not be included in
<number>).
- Phone numbers in the address book which cannot be normalized
cannot be matched.
[ 'is|contains|begins-with|ends-with', '<field>', '<text>',
'<flags>' ] - compares a specified field against the search
text. For the 'is' operation, the entire field must match, for
'contains' anywhere inside the value, for 'begins_with' at the
beginning and for 'ends_with' at the end.
Fields are referenced as in the contact dictionary (see below), using
multiple path components if necessary. Supported for matching are:
'full-name' - string
'nickname' - string
'structured-name/family' - string
'structured-name/given' - string
'structured-name/additional' - string
'phones/value' - telephone number
'emails/value' - string
'addresses/po-box' - string
'addresses/extension' - string
'addresses/street' - string
'addresses/locality' - string
'addresses/region' - string
'addresses/postal-code' - string
'addresses/country' - string
The fields referencing value lists ('phones', 'email', 'address')
check against any of the entries in these lists.
Except for 'phones/value', all values are treated as text values.
For text values, the default search without explicit flags is
very tolerant, meaning that it ignores quite a few differences
between search term and value. The default search:
- transliterates any foreign script in search term and values
to Latin before comparison, thus finding 江 when searching
for Jiang and vice-versa
- is case-insensitive
- is accent-insensitive
Case and accent differences get removed after the optional
transliteration. Spaces between words always matter.
This behavior can be modified by giving additional,
optional flags after the search value:
'case-insensitive' - force case-insensitive search (available for the sake
of consistency and just in case, should the default ever change)
'case-sensitive' - force case-sensitive search
'accent-insensitive', 'accent-sensitive' - same for accents
'transliteration' - force transliteration, i.e. explicitly choose the
current default
'no-transliteration' - disable transliteration
For telephone numbers, only digits are compared. Latin alphabetic
characters are treated as aliases for digits as they typically
occur on a keypad or old rotary dial phones ('A', 'b', 'c' map to
'1', etc.).
If the full name was not set explicitly for a contact, the
concatenation of the given, middle and family with a space as
separator is used instead when matching against the 'full-name'
field.
Using the current syntax it is not possible to define searches
where the *same* value in a value list must meet different
criteria ("cell phone number containing the digits
1234"). Something like that could be added as a future extensions,
for example by allowing search values to have more complex types
than the simple '<text>'.
[ 'any-contains', '<text>', <flags> ] - Sub-string search for
<text> in the following contact values: first, middle or last
name, formatted name, nick name, phone number, or email
address. Optional flags include: 'case-insensitive' (the default),
'case-sensitive'.
This search is equivalent to:
[ 'or',
[ 'contains', 'structured-name/given', '<text>', <flags> ],
[ 'contains', 'structured-name/additional', '<text>', <flags> ],
[ 'contains', 'structured-name/family', '<text>', <flags> ],
[ 'contains', 'full-name', '<text>', <flags> ],
[ 'contains', 'emails/value', '<text>', <flags> ],
[ 'contains', 'phones/value', '<text>']
]
Note that lookup and search are different: the former is based on a
valid number, the later on user input.
A 'phone' lookup can compare normalized numbers including the country
code, to ensure that the lookup is exact and does not mismatch numbers
from different countries. Heuristics like suffix matching do not do
this correctly in all cases.
An 'any-contains' search is based on user input, which might contain
just some digits in the middle of the phone number. The search ignores
formatting in both input and address book.
Compound searches with 'and' and 'or' are evaluated lazily, from the
first to the last sub-search. Therefore it makes sense to list
sub-searches that are more likely to match first.
Phone number lookup
===================
A "phone" search must return results quickly (<30ms with 10000
contacts) under all circumstances, including the period of time where
the unified address book is still getting assembled in memory. To
achieve this, SyncEvolution searches directly in the active address
books and presents these results until the ones from the unified
address book are ready.
A quiescence signal will be sent when:
1. results from EDS are complete before the ones from the unified
address book
2. results from the unified address book are complete.
The first signal will be skipped and the EDS results discarded if EDS
turns out to be slower than the unified address book.
Results from different EDS address books are not unified, for the sake
of simplicity. They get sorted according to the sort order that was
active when starting the search. Changing the sort order while the
search runs will only affect the final results from the unified
address book.
Refining such a search is not supported because refining a phone
number lookup is not useful.
Peers
=====
The following keys are supported for the configuration of a peer:
- "protocol" - defines how to access the address book. "PBAB" (phone
book access protocol) and "file" (read vCard files from
directory) are implemented. "SyncML" and "CardDAV"
could be added easily.
- "transport" - defines how to establish the connection. The only
supported value is "Bluetooth" (for protocol=PBAP or
SyncML), which is also the default if not given
explicitly.
- "address" - the Bluetooth MAC address in the aa:bb:cc:dd:ee:ff
format (for transport=Bluetooth).
- "database" - empty or unset for the internal address book
(protocol=PBAP), or the path to the directory
(protocol=file).
- "logdir" - a directory in which directories are created with debug
information about sync session.
- "maxsessions" - number of sessions that are allowed to exist
after a sync (>= 0): 0 is special and means unlimited,
1 for just the latest, etc.;
old sessions are pruned heuristically (for example,
keep sessions where something changed instead of
some where nothing changed), so there is no hard
guarantee that the last n sessions are present.
Not supported via the API at the moment:
- selecting a specific phone address book
- selecting which vCard properties get cached
Syncing
=======
SyncPeer() and SyncPeerWithFlags() in SyncEvolution will return a dict
with all of the following entries set:
"modified": boolean - data was modified
"added" : integer - number of new contacts
"updated" : integer - number of updated contacts
"removed" : integer - number of deleted contacts
In other words, the caller can reliably detect when nothing changed,
but when contacts were modified or added, it needs to read them to
determine which kind of properties were modified or added.
It is possible that the same contact gets counted twice, for example
when adding it in the first text-only cycle and later updating it with
photo data in the second cycle, or updating some text first and the
photo later.
Additional control over syncing is possible via the flags dictionary
in SyncPeerWithFlags(). The following keys are possible:
"progress-frequency": double - desired frequency in Hz (= events
per second) of the SyncProgress "progress" event. Default is 1/s.
"pbap-sync": string "text"|"all"|"incremental" - synchronize only
text properties, all properties or first text, then all. The default is
taken from SYNCEVOLUTION_PBAP_SYNC if set, otherwise it is "incremental".
"pbap-chunk-max-count-photo", "pbap-chunk-max-count-no-photo",
"pbap-chunk-transfer-time", "pbap-chunk-offset": int32_t -
see PBAP README.
"pbap-chunk-time-lambda": double - see PBAP README
When doing text syncing, local photo data is preserved. The
incremental sync has the same effect as a text-only sync followed by
an all-properties sync. The only difference is that it looks and
behaves like a single sync of the peer.
The SyncProgress is triggered by SyncEvolution with three different
keys (in this order, with "modified" occuring zero or more times):
"started" ("modified" | "progress")* "done"
"started" and "done" send an empty data dictionary. "modified" sends
the same dictionary as the one returned by SyncPeer(), if and only if
contact data was modified. So by definition, "modified" will be True
in the dictionary, but is included anyway for the sake of consistency.
"progress" sends a dictionary with these keys:
- "percent": double - value from 0 (start of sync) to 1 (end of sync)
- "modified": undefined value - set if the database was modified since the
last "progress" event. Detecting the exact moment when a change is made can
be hard, so the implementation may err on the side of caution and
signal a change more often than strictly necessary. The recipient must
check the database anyway to determine what changed. The value is
intentionally undefined at the moment and the recipient must not make
assumptions about it. Instead it should just check for the presence
of the entry. In the future, more specific information about what was
modified might be included as value.
- "sync-cycle": string - describes in what phase the sync is at the moment,
see "pbap-sync" in SyncPeerWithFlags()
- "text" - text-only sync active
- "all" - non-incremental text and picture sync active
- "incremental-text" - incremental sync active, currently syncing text
- "incremental-picture" - incremental sync active, done with text, now syncing pictures
- "" - some other kind of sync (non-PBAP, for example)
"progress" is guaranteed to be emitted at the end of each cycle. In
contrast to "modified", it will also be sent throughout the sync *if
and only if* progress has been made, regardless whether that progress
led to contact data changes or an increase in the completion
percentage. That way, the event also serves as a heartbeat mechanism.
The completion percentage is based purely on the number of contacts on
the peer, which is the only thing known about the peer at the start of
a sync. In a simple sync involving just one cycle, the percentage is
"number of processed contacts / total contacts".
In a sync involving two cycles, it is "number of processed contacts /
2 / total contacts". Because each contact gets processed once per
cycle, that gives 50% completion at the end of the first cycle and
100% at the end of the second.
Note that progress will not be linear over time even for a simple
sync, because some contacts will take more time to download and
process than others. For an incremental sync, the first cycle is
likely to be much faster than the second one because the second one
includes photo data.
Progress events get emitted at the time when progress is made. If that
is too close in time to the last progress event (see
"progress-frequency"), the event is skipped. Events are not simply
sent at regular intervals because that would emit events even when
syncing is not making progress and distort the timing information (for
example, 50% reached and recorded at time t, actual event with 50%
sent at t + delta). In practice, the number of progress events is
limited by various chunk sizes in the implementation (internal SyncML
message size, amount of new data that must come in via PBAP before
processing it), leading to a refresh rate during Bluetooth transfers
which is closer to every two seconds.
GetPeerStatus() returns two additional entries in the dictionary:
"progress": same dictionary as the SyncProgress "progress" value
"last-progress": double - time in seconds since the last "progress"
event. This is included to let a polling client determine whether
the sync is still alive.
Contact Data
============
A contact dictionary has the following key/value pairs. More properties
may be added later.
contact dictionary:
"id" - string, see API description
"source" - string, see API description
"full-name" - string, formatted by the user or automatically (vCard FN)
"nickname" - string
"structured-name" - name dictionary (vCard N)
"photo" - string, the URL (usually of a local file; EDS strips all inline photo
data in vCards and puts them into local files, leading to URLs like:
file:///home/user/.local/share/evolution/addressbook/system/photos/pas_id_5012983E0000065A_photo-file0.image%2Fjpeg)
"birthday" - birthday tuple
"emails" - value list with strings as value
"phones" - value list with strings as value
"urls" - value list with strings as value
"notes" - list of strings; in practice only one entry is supported
"addresses" - value list with an address dictionary as value
"roles" - list of role dictionaries
"groups" - list strings representing the names of groups the contact belongs to (CATEGORIES in vCard)
"location" - a pair of doubles, representing latitude + longitude (in this order, see GEO in vCard);
typically based on WGS84
name dictionary:
"family" - string, the last name
"given" - string, the fist name
"additional" - string, middle names
"prefixes" - string, name prefix like "Mr."
"suffixes" - string, name suffix like "Sr."
birthday tuple:
(yy, mm, dd) - integer values for year, month, and day
role dictionary:
"organisation" - main organization ("Foo ACME")
"title" - title inside that organization ("vice president")
"role" - role as part of that origanization ("adviser")
value list:
[ (value, [parameter, ...]), (value, ...) ]
value - depends on the property
parameter - a string, with values again depending on the property;
a value may have zero to n different parameters
phone parameters:
"voice"
"fax"
"car"
"cell"
"pager"
"pref"
"home"
"work"
"other" (might not be set explicitly)
address parameters:
"home"
"work"
"other" (might not be set explicitly)
url parameters:
"x-home-page"
"x-blog"
"x-free-busy" - public calendar free/busy URL
"x-video" - video chat URL
address dictionary:
"po-box" - string, post office box
"extension" - string, address extension
"street" - string, street name
"locality" - string, city name
"region" - string, area name
"postal-code" - string
"country" - string
Note: all of the strings and values above are defined by SyncEvolution
in individual-traits.cpp, except for parameter strings. Those come
directly from folks, more specifically AbstractFieldDetails:
http://telepathy.freedesktop.org/doc/folks/vala/Folks.AbstractFieldDetails.html
Here is an example contact dictionary, using Python syntax:
{
'full-name': 'John Doe',
'nickname': 'Johnny',
'groups': ['friends', 'soccer'],
'location': (30.12, -130.34),
'structured-name': {'given': 'John', 'family': 'Doe'},
'birthday': (2006, 1, 8),
'photo': 'file:///home/user/file.png',
'roles': [
{
'organisation': 'Test Inc.',
'role': 'professional test case',
'title': 'Senior Tester',
},
],
'source': [
('test-dbus-foo', luids[0])
],
'id': 'xyz',
'notes': [
'This is a test case which uses almost all Evolution fields.',
],
'emails': [
('john.doe@home.priv', ['home']),
('john.doe@other.world', ['other']),
('john.doe@work.com', ['work']),
('john.doe@yet.another.world', ['other']),
],
'phones': [
('business 1', ['voice', 'work']),
('businessfax 4', ['fax', 'work']),
('car 7', ['car']),
('home 2', ['home', 'voice']),
('homefax 5', ['fax', 'home']),
('mobile 3', ['cell']),
('pager 6', ['pager']),
('primary 8', ['pref']),
],
'addresses': [
({'country': 'New Testonia',
'locality': 'Test Megacity',
'po-box': 'Test Box #3',
'postal-code': '12347',
'region': 'Test County',
'street': 'Test Drive 3'},
[]),
({'country': 'Old Testovia',
'locality': 'Test Town',
'po-box': 'Test Box #2',
'postal-code': '12346',
'region': 'Upper Test County',
'street': 'Test Drive 2'},
['work']),
({'country': 'Testovia',
'locality': 'Test Village',
'po-box': 'Test Box #1',
'postal-code': '12345',
'region': 'Lower Test County',
'street': 'Test Drive 1'},
['home']),
],
'urls': [
('chat', ['x-video']),
('free/busy', ['x-free-busy']),
('http://john.doe.com', ['x-home-page']),
('web log', ['x-blog']),
],
}
Configuration and data handling
===============================
The configuration of peers is mapped to normal SyncEvolution
configurations. Every peer gets its own context, which contains both
the local source definition for the peer and sync plus target configs.
The special name space prefix "pim-manager-" used to identify them and
avoid potential conflicts with normal SyncEvolution configurations.
Look in ~/.config/syncevolution/pim-manager-* to find the
configuration files.
Local EDS databases are created with a fixed UID and name that also
have that prefix. One could go one step further than it is currently
done and set these databases to "disabled" in the ESourceRegistry,
which would hide them in Evolution.
The SyncEvolution command line can be used to view or manipulate
these databases:
https://syncevolution.org/wiki/item-operations
The sort order and set of active address books are stored persistently
in ~/.config/syncevolution/pim-manager.ini as:
- "sort" = same value as in the API
- "active" = space, comma or tab separated EDS source UUIDs
Usage
=====
The PIM manager is part of the syncevo-dbus-server. It can be started
automatically via D-Bus or explicitly. When started automatically, it
logs error messages to syslog.
The PIM manager starts as idle and needs to be started via the D-Bus
Start() API before it begins assembling the unified address book.
SyncEvolution installs an XDG autostart file
(/etc/xdg/autostart/syncevo-dbus-server.desktop) which, on systems
supporting that mechanism, ensures that syncevo-dbus-server is started
once. This is used for automatic syncing, which is not active when
using only the PIM Manager. Therefore syncevo-dbus-server will
terminate again after some idle period.
It's recommended to disable this mechanism or patch
syncevo-dbus-server.desktop to start the server with different
options. In particular the "--start-pim" option may be useful to
start up the UI and the server in parallel.
Here is a list of supported options:
Help Options:
-h, --help Show help options
Application Options:
-d, --duration=seconds/'unlimited' Shut down automatically
when idle for this duration (default 300 seconds)
-v, --verbosity=level Choose amount of output, 0 = no output,
1 = errors, 2 = info, 3 = debug; default is 1.
-o, --stdout Enable printing to stdout (result of operations)
and stderr (errors/info/debug).
-s, --no-syslog Disable printing to syslog.
-p, --start-pim Activate the PIM Manager (= unified address book)
immediately.
Limitations
===========
There are no hard-coded limits. In practice the number of contacts,
parallel searches, peers, etc. is limited by CPU performance, amount
of RAM and disk space.
Functionality
=============
Caching
-------
Caching of contacts is done roughly like this:
1. The PBAP backend reads all contact data. See src/backends/pbap.
2. As part of a local sync (see main SyncEvolution README), a local-cache-slow
sync is done. In that mode all local data is compared against the incoming
data. Matches are found according to the properties that are defined
as compare="always" or compare="slowsync" in src/syncevo/configs/datatypes/00vcard-fieldlist.xml.
In the default file, these are the first, middle and last name and the organization.
3. Local entries which have a match are updated with data from that match,
unmatched local entries are removed and unmatched incoming ones are
created.
This means that changing "less important" properties will be turned
into local updates. This has the desirable effect of not changing the
local ID of the existing contact, which might become important when
attaching local meta data to that contact via its local ID (not
supported at the moment; folks itself has a favorite flag which is
stored like that). It also reduces disk IO.
If matching fails, a contact will get deleted and recreated. The end result
in the unified address book is still the same, because folks does not
rely on the ID for linking.
Supported fields
----------------
The PBAP backend always downloads the entire vCard. It supports restricting
the vCard to specific properties, see src/backend/pbap/README. However, this
is not used by the PIM Manager.
In particular, PHOTO data is always included in the download right
away. A mode where caching only works without PHOTO in the initial
pass and then adds that in a second step could be added.
Internally a vCard is represented as a Synthesis field list (again,
see 00vcard-fieldlist.xml). X- extensions in the incoming vCard are
preserved.
Unified address book
--------------------
The unified address book is assembled by folks based on properties
that it considers "linkable". Linkable properties must be unique for a
person. A phone number is not linkable, because different persons
living at the same place might share a phone. For EDS, the linkable
properties are defined in folk's backends/eds/lib/edsf-persona.vala
and currently are:
- instant messaging handles
- email addresses
- local IDs (not used in this context)
- web service addresses (not used in this context)
folks supports heuristics for identifying persons which are likely to
be the same. This is not used by the PIM Manager. If it was, then links
would have to be established based on the local IDs and keeping those
stable became more important.
Localization
------------
The usual environment variables are used to determine the locale:
LC_CTYPE, LC_ALL, and LANG in that order (i.e. LC_CTYPE first and LANG
last). There is no support for changing that once syncevo-dbus-server
runs.
boost::locale is used for collation while sorting contacts. The
secondary collation level is used, which means that case, punctuation
are ignored while accents are relevant. This is hard-coded in
locale-factory-boost.cpp as DEFAULT_COLLATION_LEVEL.
Searching is either done with a strict text comparison (case
sensitive) or after using case folding (case insensitive, see
http://www.boost.org/doc/libs/1_51_0/libs/locale/doc/html/glossary.html#term_case_folding).
A 'phone' lookup uses libphonenumber to compare the free-form text
field for telephone numbers against a caller ID. The text field must
be in a format that libphonenumber understands, otherwise it will be
ignored. A country code is optional. If missing, the country code of
the current locale will be added before the comparison.
Testing
=======
Tests are mostly written in Python. See test/test-dbus.py (base
classes and testing of the normal SyncEvolution D-Bus API) and
src/dbus/server/pim/testpim.py.
These operations manipulate the system address book. To avoid
unexpected data loss when a developer runs the script in his
normal environment, check that testpim.py gets run like this:
XDG_CONFIG_HOME=`pwd`/temp-testpim/config \
XDG_DATA_HOME=`pwd`/temp-testpim/local/cache \
PATH=<SyncEvolution install path>/bin:<SyncEvolution install path>/libexec:$PATH \
<path to SyncEvolution source>/test/dbus-session.sh \
<path to SyncEvolution source>/src/dbus/server/pim/testpim.py
This will use temp-testpim in addition to temp-testdbus in the current
directory for temporary files.
If TEST_DBUS_PBAP_PHONE is set to the Bluetooth MAC address (like
A0:4E:04:1E:AD:30) of a phone, PBAP syncing with that phone is tested.
The phone must be paired, connected and support SyncML in addition to
PBAP. The test hard-codes Nokia SyncML settings to keep it simpe and
because Nokia phones are most likely to be used.
2371f69f97