Synchronizing files from one server to another is quite awesome. You can use it for backups, for keeping web servers in sync, and much more. It's fast and it doesn't take up as much bandwidth as normal copying would. And the best thing is, it can be done with only 1 command. Welcome to the wonderful world of rsync.

Installing rsync

On most modern Linux distributions you will find rsync comes preinstalled. If that's not the case, just install it with your package manager. On Ubuntu this would look like:

aptitude -y install rsync

done!

Simple - one command

Let's copy our local /home/kevin/source to /home/kevin/destination which resides on the server: server.example.com:

rsync -raz --progress --size-only /home/kevin/source/* server.example.com:/home/kevin/destination/

explained:

• -r
  recursive
• a
  archive, preserves all attributes like ownership, timestamps, etc
• z
  compress, saves bandwidth but is harder on your CPU so use it for slow/expensive connections only
• --progress
  shows you the progress of all the files that are being synced
• --size-only
  compare files based on their size instead of hashes (less CPU, so faster)

Well, that's it! But read on if you want to learn how to automate this!

Advanced - automatic syncing with SSH keys

Alright so syncing files on Linux is pretty easy. But what if we want to automate this? How can we avoid that rsync asks for a password every time?

There are different ways to go about this, but the one I mostly use is installing SSH keys. By installing your SSH key on the destination server, it will recognize you in the future and permit instant access. So this way we can automate the synchronization with rsync.

Easy script

To install your machine's SSH key to another machine, open up a terminal and download & install my instkey.bash script like this:

su -  # If you're going to use the keys to automate tasks, maybe become root first
mkdir -p ~/bin
wget -O- "http://kevin.vanzonneveld.net/download/instkey.bash/" > ~/bin/instkey.bash
chmod 755 ~/bin/instkey.bash

Now install the key on your server

open a terminal and type:

~/bin/instkey.bash server.example.com

If the script asks for a pass phrase, just hit enter twice. Then it asks for the server.example.com login. Type the password, and that's it. Your current machine can now seamlessly login to server.example.com

Did it work?

open a terminal and type:

ssh server.example.com

It should not ask you for any password. Great! this means we can also run rsync directly without logging in! If you need more in depth information on this, I wrote an article on logging in automatically with SSH keys.

Let's create a sync script

So now just create a script /root/bin/syncdata.bash

nano /root/bin/syncdata.bash

that contains your rsync command:

#!/bin/bash
rsync -raz --delete /home/kevin/source/* server.example.com:/home/kevin/destination/

Save the file (CTRL+O) and exit (CTRL+X) and make it executable like this:

chmod a+x /root/bin/syncdata.bash

Schedule it to run every hour

And to have your data synchronized every hour, open up your crontab editor:

crontab -e

And type

0 * * * * /root/bin/syncdata.bash

(if you need more in depth information on crontab I've written another article on scheduling tasks on linux using crontab)

That's it! New files are automatically updated @ server.example.com:/home/kevin/destination/ every hour. Files that are deleted from /home/kevin/source/ are also deleted at the destination, thanks to the --delete parameter. 

Some extra rsync command line options

Some extra arguments that might come in handy customizing your synchronization job:

• --delete
  delete files remotely that no longer exist locally
• --dry-run
  show what would have been transferred, but do not transfer anything
• --max-delete=10
  don't delete more than 10 files in one run, safety precaution
• --delay-updates
  put all updated files into place at transfer's end, very useful for live systems
• --compress-level=9
  explicitly set compression level 9. 0 disabled compression
• --exclude-from=/root/sync_exclude
  specifies a /root/sync_exclude that contains exclude patterns (one per line). filenames matching these patterns will not be tranfered
• --bwlimit=1024
  This option specifies a maximum transfer rate of 1024 kilobytes per second.

Pitfalls

• Of course you should really be carefull where and when to install ssh keys, because if one machine is comprimised, it's very easy for a cracker to hop to the next system without logging in. So choose wisely when to use this technology.
• Keys are user user specific. So if you're going to run programs as root that need to automatically login to systems, you must also install the key as root.