maryjane/syncevolution
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

- implemented automatic backups, logging and database comparison

Browse source 
- prepared 0.2 release


git-svn-id: https://zeitsenke.de/svn/SyncEvolution/trunk@57 15ad00c4-1369-45f4-8270-35d70d36bdcd
This commit is contained in:
Patrick Ohly 2006-03-19 21:37:30 +00:00
parent 11e4fff4a7
commit 12b98e85c4
11 changed files with 481 additions and 40 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
NEWS
View file

@ -1,3 +1,23 @@
SyncEvolution 0.2, 2006-03-19
-----------------------------
* Added automatic backup mechanism and log storage,
see "Automatic Backups and Logging".
* Output no longer is the original log data, but rather
a human-readable report of errors and synchronization
results.
* "normalize_vcard" can now also compare two .vcf files
directly.
* improved unit tests to catch more errors
* hide certain differences in vcards coming back from
the server: duplication of extended vcard properties,
missing TYPE=OTHER
* fixed client library problems:
see http://forge.objectweb.org/tracker/?group_id=96&atid=100096
#304792, #304829
* added some more problems to the "Known Problems" section
SyncEvolution 0.1, 2006-03-13
-----------------------------

95
README
View file

@ -19,7 +19,7 @@ Introduction
SyncEvolution synchronizes Evolution's contact and calender items
[calender not implemented yet]
with a SyncML server. The items are exchanged in the vCard and
vCalender formats and via the Funambol Sync4j C++ client API library,
vCalender formats via the Funambol Sync4j C++ client API library,
which should make SyncEvolution compatible with the majority of
SyncML servers. Full, one-way and incremental synchronization of items
are supported.
@ -41,15 +41,7 @@ following works:
- conflict resolution (where two clients modify the same item,
then sync with the server) is handled by the server, but
SyncEvolution has support which ensures that no data is lost
by creating duplicates (see Conflict Resolution below)
Although all of the features are covered by unit testing and
have been verified to work, it is highly recommended that you
make a backup of your
$HOME/.evolution/addressbook
$HOME/.evolution/calender
directories before running it for the first time. In older Evolution
versions the same data is found in $HOME/evolution.
by creating duplicates (see "Conflict Resolution" below)
Installation
@ -75,6 +67,14 @@ For Sync4j V2.3, an additional patch is recommended to preserve
line breaks of items on the server:
http://forge.objectweb.org/tracker/index.php?func=detail&aid=304718&group_id=96&atid=300096
Although all of the features are covered by unit testing and
have been verified to work, it is highly recommended that you
make a backup of your
$HOME/.evolution/addressbook
$HOME/.evolution/calender
directories before running it for the first time. In older Evolution
versions the same data is found in $HOME/evolution.
Usage
-----
@ -91,17 +91,24 @@ The <server> string is used to find the configuration which determines
how synchronization is going to proceed. Selection of sources of
Evolution data which are to be synchronized with that server is done
via configuration files. It is possible to configure sources without
activating their synchronization, see the "disabled" property below.
activating their synchronization, see the "disabled" property.
If the SyncML server is not specified, SyncEvolution lists all
available Evolution backend databases.
Progress and error messages are both sent to stdout. In case of an
error the synchronization run is aborted and SyncEvolution returns a
non-zero value. Recovery from failed synchronization is done by
forcing a full synchronization during the next run, i.e. by sending
all items and letting the SyncML server compare against the ones it
already knows.
Progress and error messages are written into a log file that is
preserved for each synchronization run. Details about that is found
in the "Automatic Backups and Logging" section below. Immediately
before quitting SyncEvolution will show all errors or warnings
encountered and print a summary of how the databases were modified.
This is done with the "normalize_vcard" utility script described
in the "Exchanging Data" section.
In case of an error the synchronization run is aborted prematurely and
SyncEvolution will return a non-zero value. Recovery from failed
synchronization is done by forcing a full synchronization during the
next run, i.e. by sending all items and letting the SyncML server
compare against the ones it already knows.
After a successful synchronization the server's configuration file is
updated so that the next run can be done incrementally. If the
@ -161,6 +168,49 @@ this case the directory that contains the source's config.txt should
only be accessible by the user. [NOT IMPLEMENTED YET]
Automatic Backups and Logging
-----------------------------
To support recovery from a synchronization which damaged the
local database or modified it in an unexpected way, SyncEvolution
always creates the following files during a synchronization:
- a dump of the database in a format which can be imported
back into Evolution, e.g. .vcf for address books
- a full log file with debug information
- a dump of the database after the synchronization for
automatic comparison of the before/after state with
"normalize_vcard"
If the source configuration option "logdir" is set, then
a new directory will be created for each synchronization
in that directory, using the format
SyncEvolution-<server>-<yyyy>-<mm>-<dd>-<hh>-<mm>[-<seq>]
with the various fields filled in with the time when the
synchronization started. The sequence suffix will only be
used when necessary to make the name unique. By default,
SyncEvolution will never delete any data in that log
directory unless explicitly asked to keep only a limited
number of previous log directories.
This is done by setting the "maxlogdirs" limit to something
different than the empty sring or 0: if a limit is set,
then SyncEvolution will only keep that many log directories
and start removing the oldest ones when it reaches the limit.
This cleanup is only done after a successful synchronization
and is limited to directories starting with the
SyncEvolution-<server>
prefix, so it is safe to put other files or directories
into the configured log directory.
If that option is not set, then the directory will be
created as
$TMPDIR/SyncEvolution-<username>-<server>
with access allowed for the user only. Files from a
previous synchronization will be overwritten. This is
a lot less useful because the data will probably
be lost during the next reboot.
Exchanging Data
---------------
@ -282,6 +332,9 @@ and server database that Evolution might be synchronized with.
Known Problems
--------------
ObjectWeb #<num> refers to the Funambol issue tracker at:
http://forge.objectweb.org/tracker/?group_id=96&atid=100096
- Evolution 2.0.4 and 2.4.2.1 still display the old content of a contact
which was updated during a certain test (TestEvolution::testMerge).
Exact reason unknown, needs to be investigated.
@ -294,6 +347,8 @@ Known Problems
assertion failed
- Expression: !res
ObjectWeb #304806
- various vcard and special character related problems in the
Sync4j server and client library:
TestEvolution::testVCard fails the check that items
@ -305,6 +360,12 @@ Known Problems
Characters with special meaning in XML like & < > cannot be
exchanged.
ObjectWeb #304828, #304786, #304784, #304782
- error handling could be improved
ObjectWeb #304805, #304562
- Removing a field and then synchronizing with the Sync4j server
will not remove that field on the server. The server will preserve
the old value instead.

2
configure.in
View file

@ -1,7 +1,7 @@
dnl Process this file with autoconf to produce a configure script.
AC_INIT(src/syncevolution.cpp)
AM_INIT_AUTOMAKE(syncevolution, 0.1)
AM_INIT_AUTOMAKE(syncevolution, 0.2)
AM_CONFIG_HEADER(config.h)
AC_ARG_WITH(sync4j,

13
etc/localhost_1/spds/syncml/config.txt
View file

@ -21,6 +21,19 @@ useProxy = F
# proxy URL (http://<host>:<port>)
proxyHost =
# full path to directory where automatic backups and logs
# are stored for all synchronizations; if empty, the temporary
# directory "$TMPDIR/SyncEvolution-<username>-<server>" will
# be used to keep the data of just the latest synchronization run
logdir =
# Unless this option is set, SyncEvolution will never delete
# anything in the "logdir". If set, the oldest directories and
# all their content will be removed after a successful sync
# to prevent the number of log directories from growing beyond
# the given limit.
maxlogdirs =
# used by the SyncML library internally; do not modify
begin =
end =

1
src/EvolutionContactSource.cpp
View file

@ -220,7 +220,6 @@ int EvolutionContactSource::endSync()
void EvolutionContactSource::endSyncThrow()
{
LOG.info( m_isModified ? "EvolutionContactSource: address book was modified" : "EvolutionContactSource: no modifications" );
if (m_isModified) {
GError *gerror = NULL;
GList *nextItem;

1
src/EvolutionContactSource.h
View file

@ -62,6 +62,7 @@ class EvolutionContactSource : public EvolutionSyncSource
virtual void open();
virtual void close();
virtual void exportData(ostream &out);
virtual string fileSuffix() { return "vcf"; }
virtual SyncItem *createItem( const string &uid, SyncState state );

343
src/EvolutionSyncClient.cpp
View file

@ -20,17 +20,29 @@
#include "EvolutionSyncSource.h"
#include <spdm/DMTree.h>
#include <posix/base/posixlog.h>
#include <list>
#include <memory>
#include <vector>
#include <sstream>
#include <fstream>
#include <iomanip>
#include <iostream>
using namespace std;
#include <sys/stat.h>
#include <pwd.h>
#include <unistd.h>
#include <dirent.h>
#include <errno.h>
EvolutionSyncClient::EvolutionSyncClient(const string &server) :
m_client(Sync4jClient::getInstance()),
m_server(server),
m_configPath(string("evolution/") + server)
{
LOG.setLevel(LOG_LEVEL_INFO);
m_client.setDMConfig(m_configPath.c_str());
}
@ -39,18 +51,318 @@ EvolutionSyncClient::~EvolutionSyncClient()
Sync4jClient::dispose();
}
void EvolutionSyncClient::sync( SyncMode syncMode )
/// remove all files in the given directory and the directory itself
static void rmBackupDir(const string &dirname)
{
class sourcelist : public list<EvolutionSyncSource *> {
public:
~sourcelist() {
for( iterator it = begin();
it != end();
++it ) {
delete *it;
DIR *dir = opendir(dirname.c_str());
if (!dir) {
throw dirname + ": " + strerror(errno);
}
vector<string> entries;
struct dirent *entry;
while ((entry = readdir(dir)) != NULL) {
entries.push_back(entry->d_name);
}
closedir(dir);
for (vector<string>::iterator it = entries.begin();
it != entries.end();
++it) {
string path = dirname + "/" + *it;
if (unlink(path.c_str())
&& errno != ENOENT
#ifdef EISDIR
&& errno != EISDIR
#endif
) {
throw path + ": " + strerror(errno);
}
}
if (rmdir(dirname.c_str())) {
throw dirname + ": " + strerror(errno);
}
}
// this class owns the logging directory and is responsible
// for redirecting output at the start and end of sync (even
// in case of exceptions thrown!)
class LogDir {
string m_logdir; /**< configured backup root dir, empty if none */
int m_maxlogdirs; /**< number of backup dirs to preserve, 0 if unlimited */
string m_prefix; /**< common prefix of backup dirs */
string m_path; /**< path to current logging and backup dir */
string m_logfile; /**< path to log file there */
const string &m_server; /**< name of the server for this synchronization */
public:
LogDir(const string &server) : m_server(server) {}
// setup log directory and redirect logging into it
// @param path path to configured backup directy, NULL if defaulting to /tmp
// @param maxlogdirs number of backup dirs to preserve in path, 0 if unlimited
void setLogdir(const char *path, int maxlogdirs) {
m_maxlogdirs = maxlogdirs;
if (path && path[0]) {
m_logdir = path;
// create unique directory name in the given directory
time_t ts = time(NULL);
struct tm *tm = localtime(&ts);
stringstream base;
// SyncEvolution-<server>-<yyyy>-<mm>-<dd>-<hh>-<mm>
m_prefix = "SyncEvolution-";
m_prefix += m_server;
base << path << "/"
<< m_prefix
<< "-"
<< setfill('0')
<< setw(4) << tm->tm_year + 1900 << "-"
<< setw(2) << tm->tm_mon << "-"
<< tm->tm_mday << "-"
<< tm->tm_hour << "-"
<< tm->tm_min;
int seq = 0;
while (true) {
stringstream path;
path << base.str();
if (seq) {
path << "-" << seq;
}
m_path = path.str();
if (!mkdir(m_path.c_str(), S_IRWXU)) {
break;
}
if (errno != EEXIST) {
throw m_path + ": " + strerror(errno);
}
seq++;
}
} else {
// create temporary directory: $TMPDIR/SyncEvolution-<username>
stringstream path;
char *tmp = getenv("TMPDIR");
if (tmp) {
path << tmp;
} else {
path << "/tmp";
}
path << "/SyncEvolution-";
struct passwd *user = getpwuid(getuid());
if (user && user->pw_name) {
path << user->pw_name;
} else {
path << getuid();
}
path << "-" << m_server;
m_path = path.str();
if (mkdir(m_path.c_str(), S_IRWXU)) {
if (errno != EEXIST) {
throw m_path + ": " + strerror(errno);
}
}
}
} sources;
// redirect logging into that directory, including stderr,
// after truncating it
m_logfile = m_path + "/client.log";
ofstream out;
out.exceptions(ios_base::badbit|ios_base::failbit|ios_base::eofbit);
out.open(m_logfile.c_str());
out.close();
setLogFile(m_logfile.c_str(), true);
LOG.setLevel(LOG_LEVEL_DEBUG);
}
// return log directory, empty if not enabled
const string &getLogdir() {
return m_path;
}
// return log file, empty if not enabled
const string &getLogfile() {
return m_logfile;
}
// remove oldest backup dirs if exceeding limit
void expire() {
if (m_logdir.size() && m_maxlogdirs > 0 ) {
DIR *dir = opendir(m_logdir.c_str());
if (!dir) {
throw m_logdir + ": " + strerror(errno);
}
vector<string> entries;
struct dirent *entry;
while ((entry = readdir(dir)) != NULL) {
if (strlen(entry->d_name) >= m_prefix.size() &&
!m_prefix.compare(0, m_prefix.size(), entry->d_name, m_prefix.size())) {
entries.push_back(entry->d_name);
}
}
closedir(dir);
sort(entries.begin(), entries.end());
int deleted = 0;
for (vector<string>::iterator it = entries.begin();
it != entries.end() && entries.size() - deleted > m_maxlogdirs;
++it, ++deleted) {
string path = m_logdir + "/" + *it;
string msg = "removing " + path;
LOG.info(msg.c_str());
rmBackupDir(path);
}
}
}
// remove redirection of stderr and (optionally) also of logging
void restore(bool all) {
if (all) {
setLogFile("-", false);
LOG.setLevel(LOG_LEVEL_INFO);
} else {
setLogFile(m_logfile.c_str(), false);
}
}
~LogDir() {
restore(true);
}
};
// this class owns the sync sources and (together with
// a logdir) handles writing of per-sync files as well
// as the final report (
class SourceList : public list<EvolutionSyncSource *> {
LogDir m_logdir; /**< our logging directory */
bool m_doLogging; /**< true iff additional files are to be written during sync */
bool m_reportTodo; /**< true if syncDone() shall print a final report */
string databaseName(EvolutionSyncSource &source, const string suffix) {
return m_logdir.getLogdir() + "/" +
source.getName() + "." + suffix + "." +
source.fileSuffix();
}
void dumpDatabases(const string &suffix) {
ofstream out;
out.exceptions(ios_base::badbit|ios_base::failbit|ios_base::eofbit);
for( iterator it = begin();
it != end();
++it ) {
string file = databaseName(**it, suffix);
out.open(file.c_str());
(*it)->exportData(out);
out.close();
}
}
public:
SourceList(const string &server, bool doLogging) :
m_logdir(server),
m_doLogging(doLogging),
m_reportTodo(false) {
}
// call as soon as logdir settings are known
void setLogdir(const char *logDirPath, int maxlogdirs) {
if (m_doLogging) {
m_logdir.setLogdir(logDirPath, maxlogdirs);
}
}
// call when all sync sources are ready to dump
// pre-sync databases
void syncPrepare() {
if (m_doLogging) {
m_reportTodo = true;
// dump initial databases
dumpDatabases("before");
}
}
// call at the end of a sync with success == true
// if all went well to print report
void syncDone(bool success) {
if (m_doLogging) {
// ensure that stderr is seen again
m_logdir.restore(false);
if (m_reportTodo) {
// haven't looked at result of sync yet;
// don't do it again
m_reportTodo = false;
// dump datatbase after sync
dumpDatabases("after");
// scan for error messages
ifstream in;
in.open(m_logdir.getLogfile().c_str());
while (in.good()) {
string line;
getline(in, line);
if (line.find("[ERROR]") != line.npos) {
success = false;
cout << line << "\n";
}
}
in.close();
cout << flush;
cout << "\n";
if (success) {
cout << "Synchronization successful.\n";
} else {
cout << "Synchronization failed, see "
<< m_logdir.getLogdir()
<< " for details.\n";
}
// compare databases
cout << "\nModifications:\n";
for( iterator it = begin();
it != end();
++it ) {
cout << "*** " << (*it)->getName() << " ***\n" << flush;
string before = databaseName(**it, "before");
string after = databaseName(**it, "after");
string cmd = string("normalize_vcard '" ) +
before + "' '" + after +
"' && echo 'no changes'";
system(cmd.c_str());
}
cout << "\n";
if (success) {
m_logdir.expire();
}
}
}
}
~SourceList() {
// if we get here without a previous report,
// something went wrong
syncDone(false);
// free sync sources
for( iterator it = begin();
it != end();
++it ) {
delete *it;
}
}
};
void EvolutionSyncClient::sync(SyncMode syncMode, bool doLogging)
{
SourceList sources(m_server, doLogging);
DMTree config(m_configPath.c_str());
// find server URL (part of change id)
@ -58,6 +370,10 @@ void EvolutionSyncClient::sync( SyncMode syncMode )
auto_ptr<ManagementNode> serverNode(config.getManagementNode(serverPath.c_str()));
string url = EvolutionSyncSource::getPropertyValue(*serverNode, "syncURL");
// redirect logging as soon as possible
sources.setLogdir(serverNode->getPropertyValue("logdir"),
atoi(serverNode->getPropertyValue("maxlogdirs")));
// find sources
string sourcesPath = m_configPath + "/spds/sources";
auto_ptr<ManagementNode> sourcesNode(config.getManagementNode(sourcesPath.c_str()));
@ -103,10 +419,12 @@ void EvolutionSyncClient::sync( SyncMode syncMode )
}
if (!sources.size()) {
LOG.info( "no sources configured, done" );
return;
throw string("no sources configured");
}
// ready to go: dump initial databases and prepare for final report
sources.syncPrepare();
// build array as sync wants it, then sync
// (no exceptions allowed here)
SyncSource **sourceArray = new SyncSource *[sources.size() + 1];
@ -134,4 +452,7 @@ void EvolutionSyncClient::sync( SyncMode syncMode )
}
throw res;
}
// all went well: print final report before cleaning up
sources.syncDone(true);
}

7
src/EvolutionSyncClient.h
View file

@ -45,10 +45,13 @@ class EvolutionSyncClient {
~EvolutionSyncClient();
/**
* executes the sync, throws an exception in case of failure
* Executes the sync, throws an exception in case of failure.
* Handles automatic backups and report generation.
*
* @param syncMode setting this overrides the sync mode from the config
* @param doLogging write additional log and datatbase files about the sync
*/
void sync(SyncMode syncMode = SYNC_NONE);
void sync(SyncMode syncMode = SYNC_NONE, bool doLogging = false);
};
#endif // INCL_EVOLUTIONSYNCCLIENT

6
src/EvolutionSyncSource.h
View file

@ -122,7 +122,11 @@ class EvolutionSyncSource : public SyncSource
* Dump all data from source unmodified into the given stream.
*/
virtual void exportData(ostream &out) = 0;
/**
* file suffix for database files
*/
virtual string fileSuffix() = 0;
/**
* resets the lists of all/new/updated/deleted items

4
src/Makefile.am
View file

@ -7,6 +7,9 @@ bin_PROGRAMS = syncevolution
bin_SCRIPTS = normalize_vcard
EXTRA_DIST = normalize_vcard.pl
DISTCLEANFILES = normalize_vcard
MAINTAINERCLEANFILES = Makefile.in
normalize_vcard : normalize_vcard.pl
cp $< $@
chmod u+x $@
@ -51,7 +54,6 @@ test_LDADD = $(CORE_LDADD)
SYNC4JSRC = @SYNC4JSRC@
SYNC4J_SUBDIR = @SYNC4J_SUBDIR@
BUILT_SOURCES = $(SYNC4J_SUBDIR)/all
MAINTAINERCLEANFILES = Makefile.in
clean distclean mostlyclean distdir maintainer-clean : % : $(SYNC4J_SUBDIR)/%
clean : testclean

29
src/syncevolution.cpp
View file

@ -22,6 +22,8 @@
#include <iostream>
using namespace std;
#include <libgen.h>
#include "EvolutionContactSource.h"
#include "EvolutionSyncClient.h"
@ -44,22 +46,36 @@ int main( int argc, char **argv )
{
setLogFile("-");
LOG.reset();
LOG.setLevel(LOG_LEVEL_DEBUG);
LOG.setLevel(LOG_LEVEL_INFO);
resetError();
// Expand PATH to cover the directory we were started from?
// This might be needed to find normalize_vcard.
char *exe = strdup(argv[0]);
if (strchr(exe, '/') ) {
char *dir = dirname(exe);
string path;
char *oldpath = getenv("PATH");
if (oldpath) {
path += oldpath;
path += ":";
}
path += dir;
setenv("PATH", path.c_str(), 1);
}
free(exe);
try {
if ( argc != 2 ) {
EvolutionContactSource contactSource( string( "list" ) );
listSources( contactSource, "address books" );
fprintf( stderr, "usage: %s <server>\n", argv[0] );
fprintf( stderr, "\nusage: %s <server>\n", argv[0] );
} else {
EvolutionSyncClient client(argv[1]);
client.sync();
client.sync(SYNC_NONE, true);
}
LOG.info( "synchronization successful" );
return 0;
} catch ( int sync4jerror ) {
LOG.error( lastErrorMsg );
@ -70,10 +86,11 @@ int main( int argc, char **argv )
LOG.error( errmsg );
} catch ( const string error ) {
LOG.error( error.c_str() );
} catch ( std::ios_base::failure error ) {
LOG.error( error.what() );
} catch (...) {
LOG.error( "unknown error" );
}
LOG.error( "synchronization failed" );
return 1;
}