X-Git-Url: https://mattmccutchen.net/rsync/rsync.git/blobdiff_plain/f7731f1fc2ad065a975b5b454eb4c120bd2372b4..0417c34e2d641cbac292ba5cf8a619249c87d4e3:/TODO diff --git a/TODO b/TODO index e876ee57..3d96b7bb 100644 --- a/TODO +++ b/TODO @@ -1,143 +1,86 @@ -*- indented-text -*- BUGS --------------------------------------------------------------- +Do not rely on having a group called "nobody" -There seems to be a bug with hardlinks - - mbp/2 build$ ls -l /tmp/a /tmp/b -i - /tmp/a: - total 32 - 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 - 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 - 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 - 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 - 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 - 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 - 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 - 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 - - /tmp/b: - total 32 - 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 - 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 - 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 - 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 - 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 - 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 - 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 - 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 - mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b - building file list ... done - created directory /tmp/b - ./ - a1 - a4 - a2 => a1 - a3 => a2 - wrote 350 bytes read 52 bytes 804.00 bytes/sec - total size is 232 speedup is 0.58 - mbp/2 build$ rm -r /tmp/b - mbp/2 build$ ls -l /tmp/b - ls: /tmp/b: No such file or directory - mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b - rm: cannot remove `/tmp/b': No such file or directory - mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b - building file list ... done - created directory /tmp/b - ./ - a1 - a4 - a2 => a1 - a3 => a2 - wrote 350 bytes read 52 bytes 804.00 bytes/sec - total size is 232 speedup is 0.58 - mbp/2 build$ ls -l /tmp/b - total 32 - -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 - -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 - -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 - mbp/2 build$ ls -l /tmp/a - total 32 - -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 - -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 - -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 - -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 - - -Progress indicator can produce corrupt output when transferring directories: - - main/binary-arm/ - main/binary-arm/admin/ - main/binary-arm/base/ - main/binary-arm/comm/8.56kB/s 0:00:52 - main/binary-arm/devel/ - main/binary-arm/doc/ - main/binary-arm/editors/ - main/binary-arm/electronics/s 0:00:53 - main/binary-arm/games/ - main/binary-arm/graphics/ - main/binary-arm/hamradio/ - main/binary-arm/interpreters/ - main/binary-arm/libs/6.61kB/s 0:00:54 - main/binary-arm/mail/ - main/binary-arm/math/ - main/binary-arm/misc/ - - -lchmod - I don't think we handle this properly on systems that don't have the - call. Are there any such? - - -Cross-test versions - Part of the regression suite should be making sure that we don't - break backwards compatibility: old clients vs new servers and so - on. Ideally we would test the cross product of versions. +FEATURES ------------------------------------------------------------ +Use chroot only if supported +Allow supplementary groups in rsyncd.conf 2002/04/09 +Handling IPv6 on old machines +Other IPv6 stuff: +Add ACL support 2001/12/02 +Lazy directory creation +proxy authentication 2002/01/23 +SOCKS 2002/01/23 +FAT support +Allow forcing arbitrary permissions 2002/03/12 +--diff david.e.sewell 2002/03/15 +Add daemon --no-fork option +Create more granular verbosity jw 2003/05/15 - It might be sufficient to test downloads from well-known public - rsync servers running different versions of rsync. This will give - some testing and also be the most common case for having different - versions and not being able to upgrade. +DOCUMENTATION -------------------------------------------------------- +Keep list of open issues and todos on the web site +Perhaps redo manual as SGML -Do not rely on having a group called "nobody" +LOGGING -------------------------------------------------------------- +Memory accounting +Improve error messages +Better statistics: Rasmus 2002/03/08 +Perhaps flush stdout like syslog +Log deamon sessions that just list modules +Log child death on signal +Log errors with function that reports process of origin +verbose output David Stein 2001/12/20 +internationalization - http://www.linuxbase.org/spec/refspecs/LSB_1.1.0/gLSB/usernames.html +DEVELOPMENT -------------------------------------------------------- +Handling duplicate names +Use generic zlib 2002/02/25 +TDB: 2002/03/12 +Splint 2002/03/12 - On Debian it's "nogroup" +PERFORMANCE ---------------------------------------------------------- +File list structure in memory +Traverse just one directory at a time +Allow skipping MD4 file_sum 2002/04/08 +Accelerate MD4 -DAEMON -------------------------------------------------------------- +TESTING -------------------------------------------------------------- +Torture test +Cross-test versions 2001/08/22 +Test on kernel source +Test large files +Create mutator program for testing +Create configure option to enable dangerous tests +If tests are skipped, say why. +Test daemon feature to disallow particular options. +Create pipe program for testing +Create test makefile target for some tests -server-imposed bandwidth limits +RELATED PROJECTS ----------------------------------------------------- +rsyncsh +http://rsync.samba.org/rsync-and-debian/ +rsyncable gzip patch +rsyncsplit as alternative to real integration with gzip? +reverse rsync over HTTP Range -rsyncd over ssh - There are already some patches to do this. - BitKeeper uses a server whose login shell is set to bkd. That's - probably a reasonable approach. +BUGS --------------------------------------------------------------- -FEATURES ------------------------------------------------------------ +Do not rely on having a group called "nobody" + http://www.linuxbase.org/spec/refspecs/LSB_1.1.0/gLSB/usernames.html ---dry-run is too dry + On Debian it's "nogroup" - Mark Santcroos points out that -n fails to list files which have - only metadata changes, though it probably should. + -- -- - There may be a Debian bug about this as well. +FEATURES ------------------------------------------------------------ -use chroot +Use chroot only if supported If the platform doesn't support it, then don't even try. @@ -147,208 +90,219 @@ use chroot http://lists.samba.org/pipermail/rsync/2001-August/thread.html http://lists.samba.org/pipermail/rsync/2001-September/thread.html + -- -- + -supplementary groups +Allow supplementary groups in rsyncd.conf 2002/04/09 Perhaps allow supplementary groups to be specified in rsyncd.conf; then make the first one the primary gid and all the rest be supplementary gids. + -- -- -File list structure in memory - Rather than one big array, perhaps have a tree in memory mirroring - the directory tree. +Handling IPv6 on old machines - This might make sorting much faster! (I'm not sure it's a big CPU - problem, mind you.) + The KAME IPv6 patch is nice in theory but has proved a bit of a + nightmare in practice. The basic idea of their patch is that rsync + is rewritten to use the new getaddrinfo()/getnameinfo() interface, + rather than gethostbyname()/gethostbyaddr() as in rsync 2.4.6. + Systems that don't have the new interface are handled by providing + our own implementation in lib/, which is selectively linked in. - It might also reduce memory use in storing repeated directory names --- again I'm not sure this is a problem. + The problem with this is that it is really hard to get right on + platforms that have a half-working implementation, so redefining + these functions clashes with system headers, and leaving them out + breaks. This affects at least OSF/1, RedHat 5, and Cobalt, which + are moderately improtant. -Performance + Perhaps the simplest solution would be to have two different files + implementing the same interface, and choose either the new or the + old API. This is probably necessary for systems that e.g. have + IPv6, but gethostbyaddr() can't handle it. The Linux manpage claims + this is currently the case. - Traverse just one directory at a time. Tridge says it's possible. + In fact, our internal sockets interface (things like + open_socket_out(), etc) is much narrower than the getaddrinfo() + interface, and so probably simpler to get right. In addition, the + old code is known to work well on old machines. - At the moment rsync reads the whole file list into memory at the - start, which makes us use a lot of memory and also not pipeline - network access as much as we could. + We could drop the rather large lib/getaddrinfo files. + -- -- -Handling duplicate names - We need to be careful of duplicate names getting into the file list. - See clean_flist(). This could happen if multiple arguments include - the same file. Bad. +Other IPv6 stuff: + + Implement suggestions from http://www.kame.net/newsletter/19980604/ + and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt - I think duplicates are only a problem if they're both flowing - through the pipeline at the same time. For example we might have - updated the first occurrence after reading the checksums for the - second. So possibly we just need to make sure that we don't have - both in the pipeline at the same time. + If a host has multiple addresses, then listen try to connect to all + in order until we get through. (getaddrinfo may return multiple + addresses.) This is kind of implemented already. - Possibly if we did one directory at a time that would be sufficient. + Possibly also when starting as a server we may need to listen on + multiple passive addresses. This might be a bit harder, because we + may need to select on all of them. Hm. - Alternatively we could pre-process the arguments to make sure no - duplicates will ever be inserted. There could be some bad cases - when we're collapsing symlinks. + -- -- - We could have a hash table. - The root of the problem is that we do not want more than one file - list entry referring to the same file. At first glance there are - several ways this could happen: symlinks, hardlinks, and repeated - names on the command line. +Add ACL support 2001/12/02 - If names are repeated on the command line, they may be present in - different forms, perhaps by traversing directory paths in different - ways, traversing paths including symlinks. Also we need to allow - for expansion of globs by rsync. + Transfer ACLs. Need to think of a standard representation. + Probably better not to even try to convert between NT and POSIX. + Possibly can share some code with Samba. + NOTE: there is a patch that implements this in the "patches" subdir. - At the moment, clean_flist() requires having the entire file list in - memory. Duplicate names are detected just by a string comparison. + -- -- - We don't need to worry about hard links causing duplicates because - files are never updated in place. Similarly for symlinks. - I think even if we're using a different symlink mode we don't need - to worry. +Lazy directory creation - Unless we're really clever this will introduce a protocol - incompatibility, so we need to be able to accept the old format as - well. + With the current common --include '*/' --exclude '*' pattern, people + can end up with many empty directories. We might avoid this by + lazily creating such directories. + -- -- -Memory accounting - At exit, show how much memory was used for the file list, etc. +proxy authentication 2002/01/23 - Also we do a wierd exponential-growth allocation in flist.c. I'm - not sure this makes sense with modern mallocs. At any rate it will - make us allocate a huge amount of memory for large file lists. + Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do + HTTP Basic Proxy-Authentication. + Multiple schemes are possible, up to and including the insanity that + is NTLM, but Basic probably covers most cases. -Hard-link handling + -- -- - At the moment hardlink handling is very expensive, so it's off by - default. It does not need to be so. - Since most of the solutions are rather intertwined with the file - list it is probably better to fix that first, although fixing - hardlinks is possibly simpler. +SOCKS 2002/01/23 - We can rule out hardlinked directories since they will probably - screw us up in all kinds of ways. They simply should not be used. + Add --with-socks, and then perhaps a command-line option to put them + on or off. This might be more reliable than LD_PRELOAD hacks. - At the moment rsync only cares about hardlinks to regular files. I - guess you could also use them for sockets, devices and other beasts, - but I have not seen them. + -- -- - When trying to reproduce hard links, we only need to worry about - files that have more than one name (nlinks>1 && !S_ISDIR). - The basic point of this is to discover alternate names that refer to - the same file. All operations, including creating the file and - writing modifications to it need only to be done for the first name. - For all later names, we just create the link and then leave it - alone. +FAT support - If hard links are to be preserved: + rsync to a FAT partition on a Unix machine doesn't work very well at + the moment. I think we get errors about invalid filenames and + perhaps also trying to do atomic renames. - Before the generator/receiver fork, the list of files is received - from the sender (recv_file_list), and a table for detecting hard - links is built. + I guess the code to do this is currently #ifdef'd on Windows; + perhaps we ought to intelligently fall back to it on Unix too. - The generator looks for hard links within the file list and does - not send checksums for them, though it does send other metadata. + -- -- - The sender sends the device number and inode with file entries, so - that files are uniquely identified. - The receiver goes through and creates hard links (do_hard_links) - after all data has been written, but before directory permissions - are set. +Allow forcing arbitrary permissions 2002/03/12 - At the moment device and inum are sent as 4-byte integers, which - will probably cause problems on large filesystems. On Linux the - kernel uses 64-bit ino_t's internally, and people will soon have - filesystems big enough to use them. We ought to follow NFS4 in - using 64-bit device and inode identification, perhaps with a - protocol version bump. + On 12 Mar 2002, Dave Dykstra wrote: + > If we would add an option to do that functionality, I + > would vote for one that was more general which could mask + > off any set of permission bits and possibly add any set of + > bits. Perhaps a chmod-like syntax if it could be + > implemented simply. - Once we've seen all the names for a particular file, we no longer - need to think about it and we can deallocate the memory. + I think that would be good too. For example, people uploading files + to a web server might like to say - We can also have the case where there are links to a file that are - not in the tree being transferred. There's nothing we can do about - that. Because we rename the destination into place after writing, - any hardlinks to the old file are always going to be orphaned. In - fact that is almost necessary because otherwise we'd get really - confused if we were generating checksums for one name of a file and - modifying another. + rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/ - At the moment the code seems to make a whole second copy of the file - list, which seems unnecessary. + Ideally the patch would implement as many of the gnu chmod semantics + as possible. I think the mode parser should be a separate function + that passes back something like (mask,set) description to the rest + of the program. For bonus points there would be a test case for the + parser. - We should have a test case that exercises hard links. Since it - might be hard to compare ./tls output where the inodes change we - might need a little program to check whether several names refer to - the same file. + Possibly also --chown + (Debian #23628) + NOTE: there is a patch that implements this in the "patches" subdir. -Handling IPv6 on old machines + -- -- - The KAME IPv6 patch is nice in theory but has proved a bit of a - nightmare in practice. The basic idea of their patch is that rsync - is rewritten to use the new getaddrinfo()/getnameinfo() interface, - rather than gethostbyname()/gethostbyaddr() as in rsync 2.4.6. - Systems that don't have the new interface are handled by providing - our own implementation in lib/, which is selectively linked in. - The problem with this is that it is really hard to get right on - platforms that have a half-working implementation, so redefining - these functions clashes with system headers, and leaving them out - breaks. This affects at least OSF/1, RedHat 5, and Cobalt, which - are moderately improtant. +--diff david.e.sewell 2002/03/15 - Perhaps the simplest solution would be to have two different files - implementing the same interface, and choose either the new or the - old API. This is probably necessary for systems that e.g. have - IPv6, but gethostbyaddr() can't handle it. The Linux manpage claims - this is currently the case. + Allow people to specify the diff command. (Might want to use wdiff, + gnudiff, etc.) - In fact, our internal sockets interface (things like - open_socket_out(), etc) is much narrower than the getaddrinfo() - interface, and so probably simpler to get right. In addition, the - old code is known to work well on old machines. + Just diff the temporary file with the destination file, and delete + the tmp file rather than moving it into place. - We could drop the rather large lib/getaddrinfo files. + Interaction with --partial. + Security interactions with daemon mode? -Other IPv6 stuff: - - Implement suggestions from http://www.kame.net/newsletter/19980604/ - and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt + -- -- - If a host has multiple addresses, then listen try to connect to all - in order until we get through. (getaddrinfo may return multiple - addresses.) This is kind of implemented already. - Possibly also when starting as a server we may need to listen on - multiple passive addresses. This might be a bit harder, because we - may need to select on all of them. Hm. +Add daemon --no-fork option + + Very useful for debugging. Also good when running under a + daemon-monitoring process that tries to restart the service when the + parent exits. + + -- -- + - Define a syntax for IPv6 literal addresses. Since they include - colons, they tend to break most naming systems, including ours. - Based on the HTTP IPv6 syntax, I think we should use - - rsync://[::1]/foo/bar [::1]::bar +Create more granular verbosity jw 2003/05/15 - which should just take a small change to the parser code. + Control output with the --report option. + The option takes as a single argument (no whitespace) a + comma delimited lists of keywords. -Errors + This would separate debugging from "logging" as well as + fine grained selection of statistical reporting and what + actions are logged. + + http://lists.samba.org/archive/rsync/2003-May/006059.html + + -- -- + +DOCUMENTATION -------------------------------------------------------- + + +Keep list of open issues and todos on the web site + + -- -- + + +Perhaps redo manual as SGML + + The man page is getting rather large, and there is more information + that ought to be added. + + TexInfo source is probably a dying format. + + Linuxdoc looks like the most likely contender. I know DocBook is + favoured by some people, but it's so bloody verbose, even with emacs + support. + + -- -- + +LOGGING -------------------------------------------------------------- + + +Memory accounting + + At exit, show how much memory was used for the file list, etc. + + Also we do a wierd exponential-growth allocation in flist.c. I'm + not sure this makes sense with modern mallocs. At any rate it will + make us allocate a huge amount of memory for large file lists. + + -- -- + + +Improve error messages If we hang or get SIGINT, then explain where we were up to. Perhaps have a static buffer that contains the current function name, or @@ -368,252 +322,223 @@ Errors our load? (Debian #28416) Probably fixed now, but a test case would be good. + When running as a daemon, some errors should both be returned to the + user and logged. This will make interacting with a daemon less + cryptic. -File attributes - - Transfer ACLs. Need to think of a standard representation. - Probably better not to even try to convert between NT and POSIX. - Possibly can share some code with Samba. + -- -- -Empty directories - With the current common --include '*/' --exclude '*' pattern, people - can end up with many empty directories. We might avoid this by - lazily creating such directories. +Better statistics: Rasmus 2002/03/08 + + hey, how about an rsync option that just gives you the + summary without the list of files? And perhaps gives + more information like the number of new files, number + of changed, deleted, etc. ? -zlib + + nice idea there is --stats but at the moment it's very + tridge-oriented rather than user-friendly it would be + nice to improve it that would also work well with + --dryrun - Perhaps don't use our own zlib. + -- -- - Advantages: - - - will automatically be up to date with bugfixes in zlib - - can leave it out for small rsync on e.g. recovery disks +Perhaps flush stdout like syslog - - can use a shared library + Perhaps flush stdout after each filename, so that people trying to + monitor progress in a log file can do so more easily. See + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 - - avoids people breaking rsync by trying to do this themselves and - messing up + -- -- - Should we ship zlib for systems that don't have it, or require - people to install it separately? - Apparently this will make us incompatible with versions of rsync - that use the patched version of rsync. Probably the simplest way to - do this is to just disable gzip (with a warning) when talking to old - versions. +Log deamon sessions that just list modules + At the connections that just get a list of modules are not logged, + but they should be. -logging + -- -- - Perhaps flush stdout after each filename, so that people trying to - monitor progress in a log file can do so more easily. See - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 - At the connections that just get a list of modules are not logged, - but they should be. +Log child death on signal If a child of the rsync daemon dies with a signal, we should notice that when we reap it and log a message. - Keep stderr and stdout properly separated (Debian #23626) + -- -- - After we get the @RSYNCD greeting from the server, we know it's - version but we have not yet sent the command line, so we could just - remove the -z option if the server is too old. - For ssh invocation it's not so simple, because we actually use the - command line to start the remote process. However, we only actually - do compression in token.c, and we could therefore once we discover - the remote version emit an error if it's too old. I'm not sure if - that's a good tradeoff or not. +Log errors with function that reports process of origin + Use a separate function for reporting errors; prefix it with + "rsync:" or "rsync(remote)", or perhaps even "rsync(local + generator): ". -proxy authentication + -- -- - Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do - HTTP Basic Proxy-Authentication. - Multiple schemes are possible, up to and including the insanity that - is NTLM, but Basic probably covers most cases. +verbose output David Stein 2001/12/20 + + At end of transfer, show how many files were or were not transferred + correctly. -SOCKS + -- -- - Add --with-socks, and then perhaps a command-line option to put them - on or off. This might be more reliable than LD_PRELOAD hacks. -FAT support +internationalization - rsync to a FAT partition on a Unix machine doesn't work very well at - the moment. I think we get errors about invalid filenames and - perhaps also trying to do atomic renames. + Change to using gettext(). Probably need to ship this for platforms + that don't have it. - I guess the code to do this is currently #ifdef'd on Windows; - perhaps we ought to intelligently fall back to it on Unix too. + Solicit translations. + Does anyone care? Before we bother modifying the code, we ought to + get the manual translated first, because that's possibly more useful + and at any rate demonstrates desire. -Better statistics: + -- -- - mbp: hey, how about an rsync option that just gives you the - summary without the list of files? And perhaps gives more - information like the number of new files, number of changed, - deleted, etc. ? Rasmus: nice idea there is --stats - but at the moment it's very tridge-oriented rather than - user-friendly it would be nice to improve it that would - also work well with --dryrun +DEVELOPMENT -------------------------------------------------------- -TDB: +Handling duplicate names - Rather than storing the file list in memory, store it in a TDB. + Some folks would like rsync to be deterministic in how it handles + duplicate names that come from mering multiple source directories + into a single destination directory; e.g. the last name wins. We + could do this by switching our sort algorithm to one that will + guarantee that the names won't be reordered. Alternately, we could + assign an ever-increasing number to each item as we insert it into + the list and then make sure that we leave the largest number when + cleaning the file list (see clean_flist()). Another solution would + be to add a hash table, and thus never put any duplicate names into + the file list (and bump the protocol to handle this). - This *might* make memory usage lower while building the file list. + -- -- - Hashtable lookup will mean files are not transmitted in order, - though... hm. - This would neatly eliminate one of the major post-fork shared data - structures. +Use generic zlib 2002/02/25 + Perhaps don't use our own zlib. -chmod: + Advantages: + + - will automatically be up to date with bugfixes in zlib - On 12 Mar 2002, Dave Dykstra wrote: > If we -would add an option to do that functionality, I would vote for one > -that was more general which could mask off any set of permission bits -and > possibly add any set of bits. Perhaps a chmod-like syntax if it -could be > implemented simply. + - can leave it out for small rsync on e.g. recovery disks - I think that would be good too. For example, people uploading files - to a web server might like to say + - can use a shared library - rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/ + - avoids people breaking rsync by trying to do this themselves and + messing up - Ideally the patch would implement as many of the gnu chmod semantics - as possible. I think the mode parser should be a separate function - that passes back something like (mask,set) description to the rest - of the program. For bonus points there would be a test case for the - parser. + Should we ship zlib for systems that don't have it, or require + people to install it separately? - Possibly also --chown + Apparently this will make us incompatible with versions of rsync + that use the patched version of rsync. Probably the simplest way to + do this is to just disable gzip (with a warning) when talking to old + versions. - (Debian #23628) + -- -- ---diff +TDB: 2002/03/12 - Allow people to specify the diff command. (Might want to use wdiff, - gnudiff, etc.) + Rather than storing the file list in memory, store it in a TDB. - Just diff the temporary file with the destination file, and delete - the tmp file rather than moving it into place. + This *might* make memory usage lower while building the file list. - Interaction with --partial. + Hashtable lookup will mean files are not transmitted in order, + though... hm. - Security interactions with daemon mode? + This would neatly eliminate one of the major post-fork shared data + structures. - (Suggestion from david.e.sewell) + -- -- -Incorrect timestamps (Debian #100295) +Splint 2002/03/12 - A bit hard to believe, but apparently it happens. + Build rsync with SPLINT to try to find security holes. Add + annotations as necessary. Keep track of the number of warnings + found initially, and see how many of them are real bugs, or real + security bugs. Knowing the percentage of likely hits would be + really interesting for other projects. + -- -- -Check "refuse options works" +PERFORMANCE ---------------------------------------------------------- - We need a test case for this... +File list structure in memory - Was this broken when we changed to popt? + Rather than one big array, perhaps have a tree in memory mirroring + the directory tree. + This might make sorting much faster! (I'm not sure it's a big CPU + problem, mind you.) -PERFORMANCE ---------------------------------------------------------- + It might also reduce memory use in storing repeated directory names + -- again I'm not sure this is a problem. -MD4 file_sum + -- -- - If we're doing a local transfer, or using -W, then perhaps don't - send the file checksum. If we're doing a local transfer, then - calculating MD4 checksums uses 90% of CPU and is unlikely to be - useful. - Indeed for transfers over zlib or ssh we can also rely on the - transport to have quite strong protection against corruption. +Traverse just one directory at a time - Perhaps we should have an option to disable this, analogous to ---whole-file, although it would default to disabled. The file -checksum takes up a definite space in the protocol -- we can either -set it to 0, or perhaps just leave it out. + Traverse just one directory at a time. Tridge says it's possible. -MD4 + At the moment rsync reads the whole file list into memory at the + start, which makes us use a lot of memory and also not pipeline + network access as much as we could. - Perhaps borrow an assembler MD4 from someone? + -- -- - Make sure we call MD4 with properly-sized blocks whenever possible - to avoid copying into the residue region? -String area code +Allow skipping MD4 file_sum 2002/04/08 - Test whether this is actually faster than just using malloc(). If - it's not (anymore), throw it out. - + If we're doing a local transfer, or using -W, then perhaps don't + send the file checksum. If we're doing a local transfer, then + calculating MD4 checksums uses 90% of CPU and is unlikely to be + useful. -PLATFORMS ------------------------------------------------------------ + We should not allow it to be disabled separately from -W, though + as it is the only thing that lets us know when the rsync algorithm + got out of sync and messed the file up (i.e. if the basis file + changed between checksum generation and reception). -Win32 + -- -- - Don't detach, because this messes up --srvany. - http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html +Accelerate MD4 + Perhaps borrow an assembler MD4 from someone? -DEVELOPMENT ---------------------------------------------------------- + Make sure we call MD4 with properly-sized blocks whenever possible + to avoid copying into the residue region? -Splint + -- -- - Build rsync with SPLINT to try to find security holes. Add - annotations as necessary. Keep track of the number of warnings - found initially, and see how many of them are real bugs, or real - security bugs. Knowing the percentage of likely hits would be - really interesting for other projects. +TESTING -------------------------------------------------------------- Torture test Something that just keeps running rsync continuously over a data set likely to generate problems. -Cross-testing + -- -- - Run current rsync versions against significant past releases. - -Memory debugger - jra recommends Valgrind: +Cross-test versions 2001/08/22 - http://devel-home.kde.org/~sewardj/ + Part of the regression suite should be making sure that we + don't break backwards compatibility: old clients vs new + servers and so on. Ideally we would test both up and down + from the current release to all old versions. -Release script - - Update spec files - - Build tar file; upload - - Send announcement to mailing list and c.o.l.a. - - Make freshmeat announcement - - Update web site - - - -TESTING -------------------------------------------------------------- - -Cross-test versions - - Part of the regression suite should be making sure that we don't - break backwards compatibility: old clients vs new servers and so on. - Ideally we would test both up and down from the current release to - all old versions. + Run current rsync versions against significant past releases. We might need to omit broken old versions, or versions in which particular functionality is broken @@ -623,6 +548,10 @@ Cross-test versions some testing and also be the most common case for having different versions and not being able to upgrade. + The new --protocol option may help in this. + + -- -- + Test on kernel source @@ -635,125 +564,55 @@ Test on kernel source Use awk to pull out the 'speedup' number for each transfer. Make sure it is >= x. + -- -- + Test large files Sparse and non-sparse -Mutator program - - Insert bytes, delete bytes, swap blocks, ... - -configure option to enable dangerous tests - -If tests are skipped, say why. - -Test daemon feature to disallow particular options. - -Pipe program that makes slow/jerky connections. - -Versions of read() and write() that corrupt the stream, or abruptly -fail - -Separate makefile target to run rough tests -- or perhaps just run -them every time? - -Test "refuse options" works - - What about for --recursive? - - If you specify an unrecognized option here, you should get an error. - - -DOCUMENTATION -------------------------------------------------------- - -Update README - -Keep list of open issues and todos on the web site - -Update web site from CVS - - -Perhaps redo manual as SGML - - The man page is getting rather large, and there is more information - that ought to be added. - - TexInfo source is probably a dying format. - - Linuxdoc looks like the most likely contender. I know DocBook is - favoured by some people, but it's so bloody verbose, even with emacs - support. - - -BUILD FARM ----------------------------------------------------------- - -Add machines - - Cygwin (on different versions of Win32?) + -- -- - HP-UX variants (via HP?) - SCO - - -LOGGING -------------------------------------------------------------- +Create mutator program for testing - Perhaps flush stdout after each filename, so that people trying to - monitor progress in a log file can do so more easily. See - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 + Insert bytes, delete bytes, swap blocks, ... - At the connections that just get a list of modules are not logged, - but they should be. + -- -- - If a child of the rsync daemon dies with a signal, we should notice - that when we reap it and log a message. - Keep stderr and stdout properly separated (Debian #23626) +Create configure option to enable dangerous tests - Use a separate function for reporting errors; prefix it with - "rsync:" or "rsync(remote)", or perhaps even "rsync(local - generator): ". + -- -- -verbose output - - Indicate whether files are new, updated, or deleted - At end of transfer, show how many files were or were not transferred - correctly. +If tests are skipped, say why. --vv + -- -- - Explain *why* every file is transferred or not (e.g. "local mtime - 123123 newer than 1283198") +Test daemon feature to disallow particular options. -debugging of daemon + -- -- - Add an rsyncd.conf parameter to turn on debugging on the server. +Create pipe program for testing + Create pipe program that makes slow/jerky connections for + testing Versions of read() and write() that corrupt the + stream, or abruptly fail -NICE ----------------------------------------------------------------- + -- -- ---no-detach and --no-fork options - Very useful for debugging. Also good when running under a - daemon-monitoring process that tries to restart the service when the - parent exits. +Create test makefile target for some tests -hang/timeout friendliness + Separate makefile target to run rough tests -- or perhaps + just run them every time? -internationalization + -- -- - Change to using gettext(). Probably need to ship this for platforms - that don't have it. - - Solicit translations. - - Does anyone care? Before we bother modifying the code, we ought to - get the manual translated first, because that's possibly more useful - and at any rate demonstrates desire. +RELATED PROJECTS ----------------------------------------------------- rsyncsh @@ -763,22 +622,37 @@ rsyncsh current host, directory and so on. We can probably even do completion of remote filenames. + -- -- -RELATED PROJECTS ----------------------------------------------------- http://rsync.samba.org/rsync-and-debian/ + + -- -- + + rsyncable gzip patch Exhaustive, tortuous testing Cleanups? + -- -- + + rsyncsplit as alternative to real integration with gzip? + -- -- + + reverse rsync over HTTP Range Goswin Brederlow suggested this on Debian; I think tridge and I talked about it previous in relation to rproxy. + Addendum: It looks like someone is working on a version of this: + + http://zsync.moria.org.uk/ + + -- --