--- /dev/null
+-*- indented-text -*-
+
+Notes towards a new version of rsync
+Martin Pool <mbp@samba.org>
+
+
+Good things about the current implementation:
+
+ - Widely known and adopted.
+
+ - Fast/efficient, especially for moderately small sets of files over
+ slow links (transoceanic or modem.)
+
+ - Fairly reliable.
+
+ - The choice of runnning over a plain TCP socket or tunneling over
+ ssh.
+
+ - rsync operations are idempotent: you can always run the same
+ command twice to make sure it worked properly without any fear.
+ (Are there any exceptions?)
+
+ - Small changes to files cause small deltas.
+
+ - There is a way to evolve the protocol to some extent.
+
+ - rdiff and rsync --write-batch allow generation of standalone patch
+ sets. rsync+ is pretty cheesy, though. xdelta seems cleaner.
+
+ - Process triangle is creative, but seems to provoke OS bugs.
+
+ - "Morning-after property": you don't need to know anything on the
+ local machine about the state of the remote machine, or about
+ transfers that have been done in the past.
+
+ - You can easily push or pull simply by switching the order of
+ files.
+
+
+Bad things about the current implementation:
+
+ - Persistent and hard-to-diagnose hang bugs remain
+
+ - Protocol is sketchily documented, tied to this implementation, and
+ hard to modify/extend
+
+ - Both the program and the protocol assume a single non-interactive
+ one-way transfer
+
+ - A list of all files are held in memory for the entire transfer,
+ which cripples scalability to large file trees
+
+ - Opening a new socket for every operation causes problems,
+ especially when running over SSH with password authentication.
+
+ - Renamed files are not handled: the old file is removed, and the
+ new file created from scratch.
+
+ - The versioning approach assumes that future versions of the
+ program know about all previous versions, and will do the right
+ thing.
+
+ - People always get confused about ':' vs '::'
+
+ - Error messages can be cryptic.
+
+
+Protocol philosophy:
+
+ *The* big difference between protocols like HTTP, FTP, and NFS is
+ that their fundamental operations are "read this file", "delete
+ this file", and "make this directory", whereas rsync is "make this
+ directory like this one".
+
+
+Questionable features:
+
+ These are neat, but not necessarily clean or worth preserving.
+
+ - The remote rsync can be wrapped by some other program, such as in
+ tridge's rsync-mail scripts. The general feature of sending and
+ retrieving mail over rsync is good, but this is perhaps not the
+ right way to implement it.
+
+
+Desirable features:
+
+ These don't really require architectural changes; they're just
+ something to keep in mind.
+
+ - Synchronize ACLs and extended attributes
+
+ - Anonymous servers should be efficient
+
+ - Code should be portable to non-UNIX systems
+
+ - Should be possible to document the protocol in RFC form
+
+ - --dry-run option
+
+ - IPv6 support. Pretty straightforward.
+
+ - Allow the basis and destination files to be different. For
+ example, you could use this when you have a CD-ROM and want to
+ download an updated image onto a hard drive.
+
+ - Efficiently interrupt and restart a transfer. We can write a
+ checkpoint file that says where we're up to in the filesystem.
+ Alternatively, as long as transfers are idempotent, we can just
+ restart the whole thing. [NFSv4]
+
+ - Scripting support.
+
+ - Propagate atimes and do not modify them. This is very ugly on
+ Unix. It might be better to try to add O_NOATIME to kernels, and
+ call that.
+
+ - VFS. Useful?
+
+ - Unicode. Probably just use UTF-8 for everything.
+
+
+Hard links:
+
+ At the moment, we can recreate hard links, but it's a bit
+ inefficient: it depends on holding a list of all files in the tree.
+ Every time we see a file with a linkcount >1, we need to search for
+ another known name that has the same (fsid,inum) tuple. We could do
+ that more efficiently by keeping a list of only files with
+ linkcount>1, and removing files from that list as all their names
+ become known.
+
+
+Scripting issues:
+
+ - Perhaps support multiple scripting languages: candidates include
+ Perl, Python, Tcl, Scheme (guile?), sh, ...
+
+ - Simply running a subprocess and looking at its stdout/exit code
+ might be sufficient, though it could also be pretty slow if it's
+ called often.
+
+ - There are security issues about running remote code, at least if
+ it's not running in the users own account. So we can either
+ disallow it, or use some kind of sandbox system.
+
+
+Scripting hooks:
+
+ - Whether to transfer a file
+
+ - What basis file to use
+
+ - Logging
+
+ - Whether to allow transfers (for public servers)
+
+ - Authentication
+
+ - Locking
+
+
+Interactive interface:
+
+ - Something like ncFTP, or integration into GNOME-vfs. Probably
+ hold a single socket connection open.
+
+ - Can either call us as a separate process, or as a library.
+
+ - The standalone process needs to produce output in a form easily
+ digestible by a calling program, like the --emacs feature some
+ have.
+
+ - Yow! emacs support. (You could probably build that already, of
+ course.)
+
+
+Pie-in-the-sky features:
+
+ These might have a severe impact on the protocol, and are not
+ clearly in our core requirements. It looks like in many of them
+ having scripting hooks will allow us
+
+ - Transport over UDP multicast. The hard part is handling multiple
+ destinations which have different basis files. We can look at
+ multicast-TFTP for inspiration.
+
+ - Conflict resolution. Possibly general scripting support will be
+ sufficient.
+
+ - Integrate with locking. It's hard to see a good general solution,
+ because Unix systems have several locking mechanisms, and grabbing
+ the lock from programs that don't expect it could cause deadlocks,
+ timeouts, or other problems. Scripting support might help.
+
+ - Replicate in place, rather than to a temporary file. This is
+ dangerous in the case of interruption, and it also means that the
+ delta can't refer to blocks that have already been overwritten.
+ On the other hand we could semi-trivially do this at first by
+ simply generating a delta with no copy instructions.
+
+ - Replicate block devices. Most of the difficulties here are to do
+ with replication in place, though on some systems we will also
+ have to do I/O on block boundaries.
+
+
+In favour of evolving the protocol:
+
+ - Keeping compatibility with existing rsync servers will help with
+ adoption and testing.
+
+ - We should at the very least be able to fall back to the new
+ protocol.
+
+ - Error handling is not so good.
+
+
+In favour of using a new protocol:
+
+ - Maintaining compatibility might soak up development time that
+ would better go into improving a new protocol.
+
+ - If we start from scratch, it can be documented as we go, and we
+ can avoid design decisions that make the protocol complex or
+ implementation-bound.
+
+
+Error handling:
+
+ - Errors should come back reliably, and be clearly associated with
+ the particular file that caused the problem.
+
+ - Some errors ought to cause the whole transfer to abort; some are
+ just warnings. If any errors have occurred, then rsync ought to
+ return an error.
+
+
+Concurrency:
+
+ - We want to keep the CPU, filesystem, and network as full as
+ possible as much of the time as possible.
+
+ - We can do nonblocking network IO, but not so for disk.
+
+ - It makes sense to on the destination be generating signatures and
+ applying patches at the same time.
+
+ - Can structure this with nonblocking, threads, separate processes,
+ etc.
+
+
+Uses:
+
+ - Mirroring software distributions:
+
+ - Synchronizing laptop and desktop
+
+ - NFS filesystem migration/replication. See
+ http://www.ietf.org/proceedings/00jul/00july-133.htm#P24510_1276764
+
+ - Sync with PDA
+
+ - Network backup systems
+
+ - CVS filemover
+
+
+Conflict resolution:
+
+ - Requires application-specific knowledge. We want to provide
+ policy, rather than mechanism.
+
+ - Possibly allowing two-way migration across a single connection
+ would be useful.
+
+
+Moved files:
+
+ - There's no trivial way to detect renamed files, especially if they
+ move between directories.
+
+ - If we had a picture of the remote directory from last time on
+ either machine, then the inode numbers might give us a hint about
+ files which may have been renamed.
+
+ - Files that are renamed and not modified can be detected by
+ examining the directory listing, looking for files with the same
+ size/date as the origin.
+
+
+Filesystem migration:
+
+ The NFSv4 working group wants atomic migration. Most of the
+ responsibility for this lies on the NFS server or OS.
+
+ If migrating a whole tree, then we could do a nearly-atomic rename
+ at the end. This ties in to having separate basis and destination
+ files.
+
+ NFSv4 probably wants to migrate file locks, but that's not really
+ our problem.
+
+
+Scalability:
+
+ We should aim to work well on machines in use in a year or two.
+ That probably means transfers of many millions of files in one
+ batch, and gigabytes or terabytes of data.
+
+ For argument's sake: at the low end, we want to sync ten files for a
+ total of 10kb across a 1kB/s link. At the high end, we want to sync
+ 1e9 files for 1TB of data across a 1GB/s link.
+
+ On the whole CPU usage is not normally a limiting factor, if only
+ because running over SSH burns a lot of cycles on encryption.
+
+
+Streaming:
+
+ A big attraction of rsync is that there are few round-trip delays:
+ basically only one to get started, and then everything is
+ pipelined. This is a problem with FTP, and NFS (at least up to
+ v3). NFSv4 can pipeline operations, but building on that is
+ probably a bit complicated.
Matt McCutchen's Web Site / rsync / rsync.git / commitdiff