X-Git-Url: https://mattmccutchen.net/rsync/rsync.git/blobdiff_plain/43a4dc1053bd3bfec67f3cc6a6fa4edc1f394a82..bde47ca7c52ac6b3e9f5b9bb0823c7b2ceb42768:/TODO diff --git a/TODO b/TODO index 42439806..79840312 100644 --- a/TODO +++ b/TODO @@ -1,12 +1,109 @@ -*- indented-text -*- -URGENT --------------------------------------------------------------- - - -IMPORTANT ------------------------------------------------------------ +BUGS --------------------------------------------------------------- + +rsync-url barfs on upload + + rsync foo rsync://localhost/transfer/ + + Fix the parser. + + +There seems to be a bug with hardlinks + + mbp/2 build$ ls -l /tmp/a /tmp/b -i + /tmp/a: + total 32 + 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + + /tmp/b: + total 32 + 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b + building file list ... done + created directory /tmp/b + ./ + a1 + a4 + a2 => a1 + a3 => a2 + wrote 350 bytes read 52 bytes 804.00 bytes/sec + total size is 232 speedup is 0.58 + mbp/2 build$ rm -r /tmp/b + mbp/2 build$ ls -l /tmp/b + ls: /tmp/b: No such file or directory + mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b + rm: cannot remove `/tmp/b': No such file or directory + mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b + building file list ... done + created directory /tmp/b + ./ + a1 + a4 + a2 => a1 + a3 => a2 + wrote 350 bytes read 52 bytes 804.00 bytes/sec + total size is 232 speedup is 0.58 + mbp/2 build$ ls -l /tmp/b + total 32 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + mbp/2 build$ ls -l /tmp/a + total 32 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + + +Progress indicator can produce corrupt output when transferring directories: + + main/binary-arm/ + main/binary-arm/admin/ + main/binary-arm/base/ + main/binary-arm/comm/8.56kB/s 0:00:52 + main/binary-arm/devel/ + main/binary-arm/doc/ + main/binary-arm/editors/ + main/binary-arm/electronics/s 0:00:53 + main/binary-arm/games/ + main/binary-arm/graphics/ + main/binary-arm/hamradio/ + main/binary-arm/interpreters/ + main/binary-arm/libs/6.61kB/s 0:00:54 + main/binary-arm/mail/ + main/binary-arm/math/ + main/binary-arm/misc/ + +lchmod + I don't think we handle this properly on systems that don't have the + call. Cross-test versions - Part of the regression suite should be making sure that we don't break backwards compatibility: old clients vs new servers and so on. Ideally we would test the cross product of versions. @@ -16,6 +113,39 @@ Cross-test versions some testing and also be the most common case for having different versions and not being able to upgrade. +--no-blocking-io might be broken + + in the same way as --no-whole-file; somebody needs to check. + +Do not rely on having a group called "nobody" + + http://www.linuxbase.org/spec/refspecs/LSB_1.1.0/gLSB/usernames.html + + On Debian it's "nogroup" + +DAEMON -------------------------------------------------------------- + +server-imposed bandwidth limits + +rsyncd over ssh + + There are already some patches to do this. + + BitKeeper uses a server whose login shell is set to bkd. That's + probably a reasonable approach. + + +FEATURES ------------------------------------------------------------ + + +--dry-run is insufficiently dry + + Mark Santcroos points out that -n fails to list files which have + only metadata changes, though it probably should. + + There may be a Debian bug about this as well. + + use chroot If the platform doesn't support it, then don't even try. @@ -26,12 +156,21 @@ use chroot http://lists.samba.org/pipermail/rsync/2001-August/thread.html http://lists.samba.org/pipermail/rsync/2001-September/thread.html + --files-from Avoids traversal. Better option than a pile of --include statements for people who want to generate the file list using a find(1) command or a script. + +supplementary groups + + Perhaps allow supplementary groups to be specified in rsyncd.conf; + then make the first one the primary gid and all the rest be + supplementary gids. + + File list structure in memory Rather than one big array, perhaps have a tree in memory mirroring @@ -174,6 +313,22 @@ Hard-link handling IPv6 + Perhaps put back the old socket code; if on a machine that does not + properly support the getaddrinfo API, then use it. This is probably + much simpler than reimplementing it. + + Alternatively, have two different files implementing the same + interface, and choose either the new or the old API. This is + probably necessary for systems that e.g. have IPv6, but + gethostbyaddr() can't handle it. The Linux manpage claims this is + currently the case. + + This might get us working again on RedHat 5 and similar systems. + Although the Kame patch seems like a good idea, in fact it is a much + broader interface than the relatively narrow "open by name", "accept + and log" interface that rsync uses internally, and it has the + disadvantage of clashing with half-arsed implementations of the API. + Implement suggestions from http://www.kame.net/newsletter/19980604/ and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt @@ -194,6 +349,7 @@ IPv6 which should just take a small change to the parser code. + Errors If we hang or get SIGINT, then explain where we were up to. Perhaps @@ -205,6 +361,16 @@ Errors eof" give a message that is more detailed if possible and also more helpful. + If we get an error writing to a socket, then we should perhaps + continue trying to read to see if an error message comes across + explaining why the socket is closed. I'm not sure if this would + work, but it would certainly make our messages more helpful. + + What happens if a directory is missing -x attributes. Do we lose + our load? (Debian #28416) Probably fixed now, but a test case + would be good. + + File attributes Device major/minor numbers should be at least 32 bits each. See @@ -254,6 +420,22 @@ logging At the connections that just get a list of modules are not logged, but they should be. + If a child of the rsync daemon dies with a signal, we should notice + that when we reap it and log a message. + + Keep stderr and stdout properly separated (Debian #23626) + + After we get the @RSYNCD greeting from the server, we know it's + version but we have not yet sent the command line, so we could just + remove the -z option if the server is too old. + + For ssh invocation it's not so simple, because we actually use the + command line to start the remote process. However, we only actually + do compression in token.c, and we could therefore once we discover + the remote version emit an error if it's too old. I'm not sure if + that's a good tradeoff or not. + + rsyncd over ssh There are already some patches to do this. @@ -271,6 +453,16 @@ SOCKS Add --with-socks, and then perhaps a command-line option to put them on or off. This might be more reliable than LD_PRELOAD hacks. +FAT support + + rsync to a FAT partition on a Unix machine doesn't work very well + at the moment. I think we get errors about invalid filenames and + perhaps also trying to do atomic renames. + + I guess the code to do this is currently #ifdef'd on Windows; perhaps + we ought to intelligently fall back to it on Unix too. + + Better statistics: mbp: hey, how about an rsync option that just gives you the @@ -297,6 +489,87 @@ TDB: structures. +chmod: + + On 12 Mar 2002, Dave Dykstra wrote: + > If we would add an option to do that functionality, I would vote for one + > that was more general which could mask off any set of permission bits and + > possibly add any set of bits. Perhaps a chmod-like syntax if it could be + > implemented simply. + + I think that would be good too. For example, people uploading files + to a web server might like to say + + rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/ + + Ideally the patch would implement as many of the gnu chmod semantics + as possible. I think the mode parser should be a separate function + that passes back something like (mask,set) description to the rest of + the program. For bonus points there would be a test case for the + parser. + + Possibly also --chown + + (Debian #23628) + + +--diff + + Allow people to specify the diff command. (Might want to use wdiff, + gnudiff, etc.) + + Just diff the temporary file with the destination file, and delete + the tmp file rather than moving it into place. + + Interaction with --partial. + + Security interactions with daemon mode? + + (Suggestion from david.e.sewell) + + +Incorrect timestamps (Debian #100295) + + A bit hard to believe, but apparently it happens. + + +Check "refuse options works" + + We need a test case for this... + + Was this broken when we changed to popt? + + +PERFORMANCE ---------------------------------------------------------- + +MD4 file_sum + + If we're doing a local transfer, or using -W, then perhaps don't + send the file checksum. If we're doing a local transfer, then + calculating MD4 checksums uses 90% of CPU and is unlikely to be + useful. + + Indeed for transfers over zlib or ssh we can also rely on the + transport to have quite strong protection against corruption. + + Perhaps we should have an option to disable this, analogous to + --whole-file, although it would default to disabled. The file + checksum takes up a definite space in the protocol -- we can either + set it to 0, or perhaps just leave it out. + +MD4 + + Perhaps borrow an assembler MD4 from someone? + + Make sure we call MD4 with properly-sized blocks whenever possible + to avoid copying into the residue region? + +String area code + + Test whether this is actually faster than just using malloc(). If + it's not (anymore), throw it out. + + PLATFORMS ------------------------------------------------------------ Win32 @@ -312,6 +585,7 @@ Win32 we are correct to call close(), because shutdown() discards untransmitted data. + DEVELOPMENT ---------------------------------------------------------- Splint @@ -333,14 +607,103 @@ Cross-testing Memory debugger - jra recommends: + jra recommends Valgrind: http://devel-home.kde.org/~sewardj/ +Release script + + Update spec files + + Build tar file; upload + + Send announcement to mailing list and c.o.l.a. + + Make freshmeat announcement + + Update web site + + + +TESTING -------------------------------------------------------------- + +Cross-test versions + + Part of the regression suite should be making sure that we don't + break backwards compatibility: old clients vs new servers and so + on. Ideally we would test both up and down from the current release + to all old versions. + + We might need to omit broken old versions, or versions in which + particular functionality is broken + + It might be sufficient to test downloads from well-known public + rsync servers running different versions of rsync. This will give + some testing and also be the most common case for having different + versions and not being able to upgrade. + + +Test on kernel source + + Download all versions of kernel; unpack, sync between them. Also + sync between uncompressed tarballs. Compare directories after + transfer. + + Use local mode; ssh; daemon; --whole-file and --no-whole-file. + + Use awk to pull out the 'speedup' number for each transfer. Make + sure it is >= x. + + +Test large files + + Sparse and non-sparse + +Mutator program + + Insert bytes, delete bytes, swap blocks, ... + +configure option to enable dangerous tests + +If tests are skipped, say why. + +Test daemon feature to disallow particular options. + +Pipe program that makes slow/jerky connections. + +Versions of read() and write() that corrupt the stream, or abruptly fail + +Separate makefile target to run rough tests -- or perhaps just run +them every time? + +Test "refuse options" works + + What about for --recursive? + + If you specify an unrecognized option here, you should get an error. + + DOCUMENTATION -------------------------------------------------------- Update README +Keep list of open issues and todos on the web site + +Update web site from CVS + + +Perhaps redo manual as SGML + + The man page is getting rather large, and there is more information + that ought to be added. + + TexInfo source is probably a dying format. + + Linuxdoc looks like the most likely contender. I know DocBook is + favoured by some people, but it's so bloody verbose, even with emacs + support. + + BUILD FARM ----------------------------------------------------------- Add machines @@ -353,15 +716,24 @@ Add machines SCO -NICE ----------------------------------------------------------------- ---no-detach and --no-fork options +LOGGING -------------------------------------------------------------- - Very useful for debugging. Also good when running under a - daemon-monitoring process that tries to restart the service when the - parent exits. + Perhaps flush stdout after each filename, so that people trying to + monitor progress in a log file can do so more easily. See + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 -hang/timeout friendliness + At the connections that just get a list of modules are not logged, + but they should be. + + If a child of the rsync daemon dies with a signal, we should notice + that when we reap it and log a message. + + Keep stderr and stdout properly separated (Debian #23626) + + Use a separate function for reporting errors; prefix it with + "rsync:" or "rsync(remote)", or perhaps even "rsync(local + generator): ". verbose output @@ -370,6 +742,28 @@ verbose output At end of transfer, show how many files were or were not transferred correctly. +-vv + + Explain *why* every file is transferred or not (e.g. "local mtime + 123123 newer than 1283198") + + +debugging of daemon + + Add an rsyncd.conf parameter to turn on debugging on the server. + + + +NICE ----------------------------------------------------------------- + +--no-detach and --no-fork options + + Very useful for debugging. Also good when running under a + daemon-monitoring process that tries to restart the service when the + parent exits. + +hang/timeout friendliness + internationalization Change to using gettext(). Probably need to ship this for platforms @@ -377,7 +771,9 @@ internationalization Solicit translations. - Does anyone care? + Does anyone care? Before we bother modifying the code, we ought to + get the manual translated first, because that's possibly more useful + and at any rate demonstrates desire. rsyncsh @@ -386,3 +782,21 @@ rsyncsh fairly directly into rsync commands: it just needs to remember the current host, directory and so on. We can probably even do completion of remote filenames. + + +RELATED PROJECTS ----------------------------------------------------- + +http://rsync.samba.org/rsync-and-debian/ + +rsyncable gzip patch + + Exhaustive, tortuous testing + + Cleanups? + +rsyncsplit as alternative to real integration with gzip? + +reverse rsync over HTTP Range + + Goswin Brederlow suggested this on Debian; I think tridge and I + talked about it previous in relation to rproxy.