An rsync script Part Two

March 2005

An rsync script Part Two

In the previous article, a simple rsync script for crontab and on demand use was written. At the end of the article, several possible shortcomings were mentioned.

The chosen items

The most obvious one, at least when anything that is being sent to the system scheduler, was logging. The next two are sort of ad-hoc judgment calls. A usage message (which is quite lengthy) was added simply because, well this script has been running for two months and if the user would have to rerun it by hand - frankly they wouldn't have a clue. The last item was an override for the flags and protocol in the form of additional arguments. The rsync flags are sent in via a quoted string which makes it easier to parse in the script. The reader should feel free to change the script as they see fit.

Adding the ability to do different multiple sources and/or destinations was dropped due to the amount of heads and arms the script would have to grow to facilitate the capability. Additionally, logging is simplified by invoking the script by itself for each session. Remember the golden Unix utility rule:

Do one thing and do it well

Bearing that that in mind, a multiple src/dst script would be better served with a completely different script, lest the current one morph into some sort of evil hydra.

Bit by Bit

Only portions that were added are included, however, there is a full copy [1]of the program plus a copy of the original available for anyone who would like to {ab}use it. [2]

Adding Logging

It is a shell script and as such it makes perfect sense to use the shell utility logger to access syslog directly. [3]

PROTO=ssh
RSYNC=rsync
RSYNC_FLAGS="-az --delete -e $PROTO"
LOGCMD=logger
LOGID=rsyncondemand # The name of our /var/log/messages entries
LOG_FLAGS="-t $LOGID" # Flags for the logger interface

Note that UTIL was changed to RSYNC - it is pretty much a given that rsync is being used, so for sanity it was changed. The additions here are the logger command and one flag, what the entry format will look like. Here is a sample from /var/log/messages:

Mar 16 00:32:12 pyxis syncondemand[3410]: Test

The rest

There are only three entries, however, anyone could feel free to change them. They are as follows:

for i in $PROTO $RSYNC $LOGCMD 
do
    if ! type ${i} >/dev/null; then
        $LOGCMD $LOGID "${i} not found"
        bomb "${i} not found"
    fi
done

A message in the utilities checker loop. If, for instance, the utility were installed blindly on a new machine and tossed into crontab, this rather informative message would let the administrator know that they need to install a utility or application.

$LOGCMD $LOGID "Starting $RSYNC operation from $SRC to $DST"
$RSYNC $RSYNC_FLAGS $SRC $DST
$LOGCMD $LOGID "Finished $RSYNC operation from $SRC to $DST"

A start and stop messages for recording the amount of time. What happens if they start to take extraordinary long amounts of time? These particular messages let the admin know ahead of time and can easily be added to log analysis tools.

Additional Flags & Proto

Note the top of the script with the following global definitions:

PROTO=ssh
RSYNC_FLAGS="-az --delete -e $PROTO"

In this version only those two are altered. Changing the protocol requires that the actual final rsync invocation be changed as well.

First, make the call and global different:

PROTO=ssh
RSYNC_FLAGS="-az --delete -e "
...
$RSYNC $RSYNC_FLAGS $PROTO $SRC $DST

Note that the protocol now must be defined either by the ssh default or using the modification.

Next, all that has to be done is to add a method to change the default rsync flags and protocol. It can all be done in the switch/case loop:

while getopts s:d: ch; do
    case ${ch} in
        s)
            SRC=${OPTARG} 
            ;;
        d)
            DST=${OPTARG}
            ;;
        f)
            $RSYNC_FLAGS=${OPTARG}
            ;;
        p)
            $PROTO=${OPTARG}
            ;;
    esac
done        
shift $((${OPTIND} - 1))

The only downside is that the -f has to use double quotes to replace all of the strings - regrettable - yes; but there is a usage message after all.

The Easy Part - Usage

Usage messages in shell scripts, frankly, rock. Simply because there is an all ending one simple way to do them even though they look kind of freaky relative to the indentation of the rest of the script, as usual, only the code can explain it:

usage()
{
    cat <<_usage_
Usage: ${progname} help
Usage: ${progname} [options arguments...]
Usage: ${progname} [-s[srchost:]/dir -d[dsthost:]/dir] [-p prot] [-f "flags"]
Options:
    -s SRC   Synchronize from source host:/directory or just directory
    -d DST   Synchronize to host:/directory or just directory
    -f flags Override the default rsync flags with custom flags. Must be
             enclosed in double quotes.
    -p proto Which protocol to use (rsh or ssh)
Examples:
 Sync from local directory /usr/local/mudserver to arachnos:/netbackups
     ${progname} -s /usr/local/mudserver -d arachnos:/netbackups
 Sync midge:/home to /u00/midge not using compression and remote shell
     ${progname} -s midge:/home -d /u00/midge -f "-a --delete -e rsh"
_usage_
}

Basically it is just redirection and yes, it does make a well formed script look really ugly, however, using that particular method over typing echo... echo... echo... and avoiding echo calls makes it well worth the apparent ugliness. [4]

Summary

This simple script should be a good start to organizing and managing rsync style backups. It is small, concise and easy to use or modify. Some food for thought might be adding ssh options (such as forced versions) and extended operations like cloning.

Footnotes

Browser may not understand .sh extension, save the file (it is set to 0644)
Both scripts are public domain and not licensed at all.
Logger is an incredible little utility, likely systhread will be covering it later.
Personally I think there is something aesthetic about redirection.

Previous: Rsync Script Part I