AppNote: Using RSYNC with Windows
Novell Cool Solutions: AppNote
By Gary Childers
Digg This -
Posted: 28 Apr 2005
09 Mar 2006 - Updated with new information
I have received enough replies to my previous posts on RSYNC that I thought it may be time to update this posting. Lots of people are interested in backing up data on Windows laptops and PCs using RSYNC, in a variety of situations.
RSYNC is wonderful for data synchronization, because it only copies updated data from the source to the destination, using rolling checksums on both sides to determine which files need to be updated. If the data is already there, it won't be re-copied. RSYNC also updates the data at the block level, so that only the updated parts of a file are synchronized, rather than the whole file.
For many of these situations, where you want to synchronize user data from a workstation to a NetWare server automatically, I have to say it's hard to beat Novell's iFolder application. It is easy to setup, easier to use, and it just works. For data sync over public Internet connections, iFolder automatically encrypts the transferred data using SSL (configured with a user-supplied pass phrase), as well as using regular eDirectory password login for authentication.
By default, the iFolder client runs in the background, checking your specified local folder (usually the My Documents folder) every five seconds for updated files, and checking the remote server folder every 20 seconds in case there are any updated files on that side. As soon as you update a document on the PC, as long as you have an active Internet connection and have logged into the iFolder server, you can count on an automatic backup of your files. If you connect and login later on, the file synchronization occurs at that point.
I like to use iFolder to synchronize data on two PCs (like, one at home, one at the office). I have the iFolder client configured for the same user account on both machines, so they are pointed to the same synchronization folder on the iFolder server. I update a document or create a new file on one PC and - BAM! It automatically synchronizes first to the server, then to the other PC. (My apologies to Emeril.)
iFolder is also free with NetWare 6.5 or NNLS (but a separate purchase to run on a Windows server platform - why would you do that, anyway?). iFolder is, essentially, RSYNC under the covers. It's free, it's easy, it works - who could ask for more?
Well, some folks can (and do) ask for more. Some want to have iFolder automatically sync a laptop's local data with the user's home folder on the NetWare server, so that both copies of the data are readily accessible to the user, and can be synchronized both ways. Hopefully, enough people have requested that functionality that it can be included in a future release of iFolder (cross your fingers).
Other folks are running particular (home grown) applications, and want to have more direct control over data synchronization. It is particularly these people who want more of the command-line activation and control of the RSYNC utility on the Windows platform. We can do that, too.
RSYNC for Data Backup
I am using RSYNC to backup data from Windows 2000 and XP PCs to a NetWare 6.5 server.
I have a NetWare 6.5 server with the RSYNC (2.6.0) component installed. The NetWare server runs the RSYNC daemon from SYS:\RSYNC, listening for rsync requests on default TCP port 873. I can then run NetWare Remote Manager (https://serverIP:8009) | Manage Applications | Rsync Configuration, to configure the server to accept my particular RSYNC clients.
For each Windows 2000 or XP client, create a "Branch Office" definition, select the volume to hold the replicated data, and a path, and the IP address of the XP client under "Branch Office IP Address".
You may find it easier to manually edit the SYS:\ETC\rsyncd.conf file, especially after you have added one location, so you can just copy, paste and modify additional locations (modules).
At the server console, execute rsyncstr.ncf to start the RSYNC daemon, and rsyncstp.ncf to unload it. Both are located in the SYS:\RSYNC directory (copy them to SYS:\SYSTEM).
I then downloaded RSYNC for Windows (CWRSYNC, available from http://www.itefix.no) to install the cygwin UNIX emulation system, and the RSYNC utility. The CWRSYNC package is small (2.75 MB), and installs like a Windows app to a default location of C:\cwrsync. But it really can just be copied to the hard drive, and run like a DOS app, without any installation (good old DOS!). It just finds its .DLL files from the current folder.
I've used the minimal setup for RSYNC on my XP machine, which only needs rsync.exe and cygwin1.dll (it fits on a floppy!).
The RSYNC Command
I then created a batch file (.BAT or .CMD) to rsync my data folder to the RSYNC server.
@ECHO OFF REM ********************************************* REM SYNCDATA.BAT - Batch file to test rsync command. REM ********************************************* @echo Synchronizing data files with Remote Server ... c:\cwrsync\rsync -rav /mydata RSYNC1::XPLAPTOP1 --delete --exclude-from=c:/cwrsync/excludes.txt >c:/cwrsync/rsync.log @echo Done ! pause
The bare bones of the command is simply:
c:\cwrsync\rsync -rav /mydata RSYNC1::XPLAPTOP1
RSYNC is the command. The added switches and parameters just add some functionality:
The -rav means -r for recursive, -a for archive, -v for verbose. The source string is "/mydata". The destination string is "RSYNC1::XPLAPTOP1" (the server name RSYNC1 can be resolved via the HOSTS file or DNS, and the XPLAPTOP1 corresponds to the section specified in the rsyncd.conf file on the RSYNC server). The double-colon "::" specifies that rsync communication will occur via TCP, default port 873. The --delete switch will delete files on the target that have been deleted on the source (use this carefully!). The --exclude-from switch will exclude copying files (like, *.mp3) that are listed in the specified text file. The ending switch ">c:/rsync/rsync.log" sends the verbose logging to the specified file. The final "pause" command is useful for troubleshooting. Hint: if the source string has a space in it (like My Documents), then enclose the source string in quotes (such as "/documents and settings/JoeUser/My Documents").
I also created a Windows shortcut to the batch file, and copied it to the desktop, so that all the end user has to do is double-click.
The EXCLUDES.TXT file contains file types that I don't want replicated - see the sample below.
Note that the RSYNC command will presume that the source files are on your C: drive. The RSYNC command is also case-sensitive, so pay attention to that. Also note that the slashes make a difference: the source string "/mydata" will copy the /mydata folder, and all files and sub-folders in it (assuming -r or -a is used), whereas the source string "/mydata/" (with a trailing slash) will only copy the files and sub-folders within the /mydata folder. Although the rsync command will recognize the use of the back slash (\) in certain places (such as the path for "exclude-from="), it will only recognize the use of the forward slash (/) in the source string. So get used to using forward slashes.
Other Types of Data Synchronization
If you need to synchronize files that are on drives other than C:, you can accomplish this by using a different method of specifying locations that is provided by the cwrsync implementation. We'll call this the "cygdrive" method. This places an added "/cygdrive/" in the source and/or destination string, such as:c:\cwrsync\rsync -rav /cygdrive/c/mydata RSYNC1::XPLAPTOP1
This command will perform the exact function as the (barebones) one above. However, this method allows us to specify drives other than C:, whether local or remote. For example:c:\cwrsync\rsync -rav /cygdrive/d/mydata RSYNC1::XPLAPTOP1
This is what you'd use if your source data is on your local D: drive. But it gets better. How about synchronizing data to a USB hard drive or flash media, that is currently listed as drive E:? We would use this command:c:\cwrsync\rsync -rav /cygdrive/c/mydata/ /cygdrive/e/mydata/
This performs the sync between folders on two local drives, C:\mydata\, and E:\mydata\. Note that the trailing slashes in the command will copy the contents of C:\mydata\, rather than the folder itself, and it presumes that E:\mydata already exists (if it doesn't the command will fail -- use the command without the trailing slashes if you want to create the source folder at the destination).
How about mapped drives? Absolutely. If you are one of those who need to synchronize your local documents folder with your mapped home folder on the server (say, H:), you can use this command:c:\cwrsync\rsync -rav /cygdrive/c/mydata/ /cygdrive/h/mydata/
Or if you want to pull the data in the other direction, just switch the source and destination strings, like this:c:\cwrsync\rsync -rav /cygdrive/h/mydata/ /cygdrive/c/mydata/
This would update my C:\mydata folder from a copy that is on my mapped H: drive. Usually, if I were performing this type of "pull" update (rather than the normal "push"), I would use the --update switch (as opposed to the --delete switch), to specify not to overwrite any files at the destination (C:) that are newer than at the source (H:). So the command would look like:c:\cwrsync\rsync -rav /cygdrive/h/mydata/ /cygdrive/c/mydata/ --update
Note that in the last four examples the RSYNC command is no longer being used as a client-server utility, as before, with the RSYNC daemon running on the NetWare (or Linux) server. Now we are using the RSYNC command as a simple "client-client" utility, if you will, with the local PC in control of both source and destination drives, even if one of them is mapped to a server.
On the LAN, rsync will copy 500MB of data in about 5 minutes. After that, it will only need to perform updates, adds and deletes, so it becomes very efficient. A WAN connection, such as a T1, is considerably slower (T1 = 500MB per hour). If there are firewalls in between, port 873 will have to be opened for the link.
Note: if you wish to setup an RSYNC location for a whole IP subnet of clients to access, you can set the "hosts allow" line in the rsyncd.conf file (see example below) to specify the IP subnet, instead of a single host IP address. For example, instead of:hosts allow=10.1.0.101
you can specify the subnet by using:hosts allow=10.1.0.0/24
The "/24" specifies the subnet mask (class C or 255.255.255.0) - use the one appropriate for your network IP subnet. You can also specify allowing multiple individual hosts by using:hosts allow=10.1.0.101, 10.1.0.102, 10.1.0.103
For more information on configuring the rsyncd.conf file, including using Ipv6, see: http://samba.anu.edu.au/ftp/rsync/rsyncd.conf.html
For more information on RSYNC command syntax and options, see the RSYNC man page at: http://samba.anu.edu.au/ftp/rsync/rsync.html
More Information on RSYNC
For more tips on how to use RSYNC instead of tape backup for remote site servers, see: "Using RSYNC in Data Backup Solution".For ideas on how to use RSYNC to distribute service packs or standard corporate documents, see: "Distribute Service Packs and Corporate Documents using RSYNC".
For an example of how to use RSYNC simultaneously with Nterprise Branch Office and outside of that application, and of using RSYNC in a NetWare clustered server environment, see "Double SYNC -- Using RSYNC Simultaneously With and Without Branch Office for Data Backup".
uid = nobody
gid = nobody
max connections = 0
syslog facility = local5
pid file = SYS:/rsync/rsyncd.pid
log file = SYS:/rsync/rsyncd.log
motd file = SYS:/rsync/rsyncd.motd
UPDATE - 09 Mar 2006:
Adding this parameter: "--modify-window=1" to any RSYNC command will cause it to be less precise about the file modification timestamp on all files being examined (the value is in seconds). Thus if there is a 1-second variance in the timestamps, RSYNC will assume that the files are in sync.
rsync -rav /cygdrive/c/mydata/ /cygdrive/e/mydata/ --modify-window=1
This prevents the re-transfer of identical files which may have a slight variance in the timestamp.
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com