csync2 is so cool

When you search csync2 on google, you'll get a hint:

Did you mean: sync2

That is so unfair. csync2 is one of the coolest tool to do Server Farm Synchronization among what I have tried. It solved so many problems I had before. Working as a sys admin, administrating a big server farm with hundres of linux machines, how to synchronize the data is always a big headache. You can use your own script to scp, you can use rsync, you can use subversion. Yeah, there are so many tools out there you can utilize. But, none of them are built for server synchronization.

You can also use SAN, yeah, that would be nice. But SAN solution is not cheap. You know that, right?

Ok, let's have a look at csync2. the full name is "cluster synchronization tool, 2nd generation". You can visit their website: http://oss.linbit.com/csync2/. Basicly everything you need is over there. But you'll noticed there's only one doc available: http://oss.linbit.com/csync2/paper.pdf.
The author Clifford Wolf put almost everything there in order for you to use this fantastic tool. But if you are a RedHat family user, you'll find it not that straightforward to build and finally use it.

Here is what I have done, I include a step by step buiding and configuration mini how to here. Note that I am buiding from source, since the author's rpm spec file has some problem. Maybe I can add a howto for buiding the rpm later.

For everybody who build linux from source before, the dependency hell is something always nasty. Here is a list of dependency I have encountered:

libgcrypt-1.2.2.tar.gz
libtasn1-0.3.2.tar.gz
libgpg-error-1.3.tar.gz
sqlite-2.8.17.tar.gz
gnutls-1.2.10.tar.bz2
librsync-0.9.7.tar.gz

Note that you have to use sqlite-2.x version, not the newest 3.x one.

And here is my script to buid the csync2:

cd /root/csync2
tar xvfz libgpg-error-1.3.tar.gz -C /usr/local/src/
cd /usr/local/src/libgpg-error-1.3/
./configure
make
make install
cd /root/csync2
tar xvfz libgcrypt-1.2.2.tar.gz -C /usr/local/src
cd /usr/local/src/libgcrypt-1.2.2/
./configure
make
make install
cd /root/csync2
tar xvfz libtasn1-0.3.2.tar.gz -C /usr/local/src/
cd /usr/local/src/libtasn1-0.3.2/
./configure
make
make install
cd /root/csync2
tar xvfz sqlite-2.8.17.tar.gz -C /usr/local/src/
cd /usr/local/src/sqlite-2.8.17/
./configure
make
make install
cd /root/csync2
tar xvfz librsync-0.9.7.tar.gz -C /usr/local/src/
cd /usr/local/src/librsync-0.9.7/
./configure
make
make install
cd /root/csync2
tar xvfj gnutls-1.2.10.tar.bz2 -C /usr/local/src/
cd /usr/local/src/gnutls-1.2.10/
./configure
make
make install
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig
cd /root/csync2
tar xvfz csync2-1.31.tar.gz -C /usr/local/src/
cd /usr/local/src/csync2-1.31/
./configure
make
make install
make cert
echo "csync2 30865/tcp" >> /etc/services

Notice this:
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig
This is to tell the system to add /usr/local/lib to the ld directory, which is not included by default for redhat distros. Without this, the build will keep complaining missing some dependencies but you DID install.

One advantage building from source here is the part to build a certificate, notice the line make cert;, just that, your certificate is made.

Now you have the csync2.

you can run csync2-ii on each machine to do some test. but you'd better make it an inetd service. Everyone who uses redhat knows: RedHat use xinetd to replace inetd.
so you'll need this file named csyc2 to add to your /etc/xinetd.d/

service csync2
{
disable = no
protocol = tcp
socket_type = stream
wait = no
user = root
server = /usr/local/sbin/csync2
server_args = -i
}

chkconfig xinetd on
service xinetd start

Of course you have to tell the new csync2 service by adding the port to /etc/services. That I have included in my script, it's the last line:

echo "csync2 30865/tcp" >> /etc/services

Be careful about turning on the xinetd service though. You may want to double check your /etc/xinetd.d/, make sure no other unwanted services would be turned on by this.

Now you are ready to go. Next thing you need is a good config file.

FYI: the default config file csync2 uses is /etc/csync2.cfg. Believe it or not, I didn't find it on the paper.pdf. it took me quite a while to figure out that one is the default one.

Here is a sample /etc/csync2.cfg

group serverfarm
{
host server34.domain.com;
host (server35.domain.com);
host (server36.domain.com);
host (server37.domain.com);

key /etc/serverfarm.key;

include /etc/hosts;
include /etc/csync2.cfg;
include /usr/local/apache2/conf;
include /var/www;

action
{
pattern /usr/local/apache2/conf/httpd.conf;
exec "/usr/local/apache2/bin/apachectl graceful";
logfile "/var/log/csync2_action.log";
do-local;
}

backup-directory /var/backups/csync2;
backup-generations 3;

auto none;
}

I am only synchronizing the hosts file, the csync2.cfg file, the httpd.conf file, and the server document root. Notice the backup-directory, that directory, the csync2 won't create for you, and it will complain some weird error if you don't create them by yourself. So do that now.

This line: key /etc/serverfarm.key;
that's their preshared key. generate is in any machine with:
csync2 -k /etc/serverfarm.key, and synchronize it to all your other machines. hopefully this is the last time you do a sync use your stone age solution, maybe rsync?

The coolest part IMO is the action part.
pattern /usr/local/apache2/conf/httpd.conf;
exec "/usr/local/apache2/bin/apachectl graceful";
This means whenever I change the httpd.conf file, it'll restart the apache for me. How nice!

Notice also I put () for most servers. That means, those servers are just slaves. well, it's a server farm, who's gonna change things in slaves if not for updating code? this also solves the conflict problem. Who cares who changed sth in slave machines, server34 is the one I need to put code on and propagate!


Now everything is set, ready... Go, let's make some test.

First time sync, according to the author, need to run something like -I. But in my case, I just ran a csync2 -x on server34. There are lots of conflicts, sure. but my setting is only server34 should win, so did that machine win. The first time sync make take a while, not much.

Then I added a virtual host on server34, modifed httpd.conf, added the documentroot at /var/www/dummy-host, put some files in there, then I issued again the csync2 -x, I counted, 1,2,3,4,5, when I counted to 5, the execution finished, without any message.

Then try the new virtual host name in your browser, wow, it's there, refresh, refresh, no missing server, they are all synced. only 5 seconds, 60 servers, that's easy!

Now I hope you get an idea how to use this fantastic tool. Enjoy! :)