From abb0b532f86b9db36efd463e2a744e30ac57ef36 Mon Sep 17 00:00:00 2001 From: "J.W. Schultz" Date: Wed, 17 Dec 2003 00:47:39 +0000 Subject: [PATCH] Reorganized and cleaned up TODO list. --- TODO | 1016 ++++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 625 insertions(+), 391 deletions(-) diff --git a/TODO b/TODO index e876ee57..106c9a5f 100644 --- a/TODO +++ b/TODO @@ -1,8 +1,94 @@ -*- indented-text -*- BUGS --------------------------------------------------------------- +Fix hardlink reporting 2002/03/25 +Fix progress indicator to not corrupt log +lchmod question +Do not rely on having a group called "nobody" +Incorrect timestamps (Debian #100295) +Win32 + +FEATURES ------------------------------------------------------------ +server-imposed bandwidth limits +rsyncd over ssh +Use chroot only if supported +Allow supplementary groups in rsyncd.conf 2002/04/09 +Handling IPv6 on old machines +Other IPv6 stuff: +Add ACL support 2001/12/02 +Lazy directory creation +Conditional -z for old protocols +proxy authentication 2002/01/23 +SOCKS 2002/01/23 +FAT support +Allow forcing arbitrary permissions 2002/03/12 +--diff david.e.sewell 2002/03/15 +Add daemon --no-detach and --no-fork options + +DOCUMENTATION -------------------------------------------------------- +Update README +Keep list of open issues and todos on the web site +Update web site from CVS +Perhaps redo manual as SGML + +LOGGING -------------------------------------------------------------- +Make dry run list all updates 2002/04/03 +Memory accounting +Improve error messages +Better statistics: Rasmus 2002/03/08 +Perhaps flush stdout like syslog +Log deamon sessions that just list modules +Log child death on signal +Keep stderr and stdout properly separated (Debian #23626) +Log errors with function that reports process of origin +verbose output David Stein 2001/12/20 +Add reason for transfer to file logging +debugging of daemon 2002/04/08 +internationalization + +DEVELOPMENT -------------------------------------------------------- +Handling duplicate names +Use generic zlib 2002/02/25 +TDB: 2002/03/12 +Splint 2002/03/12 +Memory debugger +Create release script +Add machines to build farm + +PERFORMANCE ---------------------------------------------------------- +File list structure in memory +Traverse just one directory at a time +Hard-link handling +Allow skipping MD4 file_sum 2002/04/08 +Accelerate MD4 +String area code + +TESTING -------------------------------------------------------------- +Torture test +Cross-test versions 2001/08/22 +Test on kernel source +Test large files +Create mutator program for testing +Create configure option to enable dangerous tests +If tests are skipped, say why. +Test daemon feature to disallow particular options. +Create pipe program for testing +Create test makefile target for some tests +Test "refuse options" works + +RELATED PROJECTS ----------------------------------------------------- +rsyncsh +http://rsync.samba.org/rsync-and-debian/ +rsyncable gzip patch +rsyncsplit as alternative to real integration with gzip? +reverse rsync over HTTP Range + -There seems to be a bug with hardlinks + +BUGS --------------------------------------------------------------- + +Fix hardlink reporting 2002/03/25 + (was: There seems to be a bug with hardlinks) mbp/2 build$ ls -l /tmp/a /tmp/b -i /tmp/a: @@ -72,8 +158,12 @@ There seems to be a bug with hardlinks -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + -- -- + -Progress indicator can produce corrupt output when transferring directories: +Fix progress indicator to not corrupt log + + Progress indicator can produce corrupt output when transferring directories: main/binary-arm/ main/binary-arm/admin/ @@ -92,21 +182,16 @@ Progress indicator can produce corrupt output when transferring directories: main/binary-arm/math/ main/binary-arm/misc/ + -- -- + + +lchmod question -lchmod I don't think we handle this properly on systems that don't have the call. Are there any such? + -- -- -Cross-test versions - Part of the regression suite should be making sure that we don't - break backwards compatibility: old clients vs new servers and so - on. Ideally we would test the cross product of versions. - - It might be sufficient to test downloads from well-known public - rsync servers running different versions of rsync. This will give - some testing and also be the most common case for having different - versions and not being able to upgrade. Do not rely on having a group called "nobody" @@ -114,187 +199,63 @@ Do not rely on having a group called "nobody" On Debian it's "nogroup" -DAEMON -------------------------------------------------------------- - -server-imposed bandwidth limits - -rsyncd over ssh - - There are already some patches to do this. - - BitKeeper uses a server whose login shell is set to bkd. That's - probably a reasonable approach. - - -FEATURES ------------------------------------------------------------ - - ---dry-run is too dry - - Mark Santcroos points out that -n fails to list files which have - only metadata changes, though it probably should. - - There may be a Debian bug about this as well. - - -use chroot - - If the platform doesn't support it, then don't even try. - - If running as non-root, then don't fail, just give a warning. - (There was a thread about this a while ago?) - - http://lists.samba.org/pipermail/rsync/2001-August/thread.html - http://lists.samba.org/pipermail/rsync/2001-September/thread.html - - -supplementary groups - - Perhaps allow supplementary groups to be specified in rsyncd.conf; - then make the first one the primary gid and all the rest be - supplementary gids. - - -File list structure in memory - - Rather than one big array, perhaps have a tree in memory mirroring - the directory tree. - - This might make sorting much faster! (I'm not sure it's a big CPU - problem, mind you.) - - It might also reduce memory use in storing repeated directory names --- again I'm not sure this is a problem. - -Performance - - Traverse just one directory at a time. Tridge says it's possible. - - At the moment rsync reads the whole file list into memory at the - start, which makes us use a lot of memory and also not pipeline - network access as much as we could. - - -Handling duplicate names - - We need to be careful of duplicate names getting into the file list. - See clean_flist(). This could happen if multiple arguments include - the same file. Bad. - - I think duplicates are only a problem if they're both flowing - through the pipeline at the same time. For example we might have - updated the first occurrence after reading the checksums for the - second. So possibly we just need to make sure that we don't have - both in the pipeline at the same time. - - Possibly if we did one directory at a time that would be sufficient. - - Alternatively we could pre-process the arguments to make sure no - duplicates will ever be inserted. There could be some bad cases - when we're collapsing symlinks. - - We could have a hash table. + -- -- - The root of the problem is that we do not want more than one file - list entry referring to the same file. At first glance there are - several ways this could happen: symlinks, hardlinks, and repeated - names on the command line. - If names are repeated on the command line, they may be present in - different forms, perhaps by traversing directory paths in different - ways, traversing paths including symlinks. Also we need to allow - for expansion of globs by rsync. +Incorrect timestamps (Debian #100295) - At the moment, clean_flist() requires having the entire file list in - memory. Duplicate names are detected just by a string comparison. + A bit hard to believe, but apparently it happens. - We don't need to worry about hard links causing duplicates because - files are never updated in place. Similarly for symlinks. + -- -- - I think even if we're using a different symlink mode we don't need - to worry. - Unless we're really clever this will introduce a protocol - incompatibility, so we need to be able to accept the old format as - well. +Win32 + Don't detach, because this messes up --srvany. -Memory accounting + http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html - At exit, show how much memory was used for the file list, etc. - Also we do a wierd exponential-growth allocation in flist.c. I'm - not sure this makes sense with modern mallocs. At any rate it will - make us allocate a huge amount of memory for large file lists. + -- -- -Hard-link handling +FEATURES ------------------------------------------------------------ - At the moment hardlink handling is very expensive, so it's off by - default. It does not need to be so. +server-imposed bandwidth limits - Since most of the solutions are rather intertwined with the file - list it is probably better to fix that first, although fixing - hardlinks is possibly simpler. + -- -- - We can rule out hardlinked directories since they will probably - screw us up in all kinds of ways. They simply should not be used. - At the moment rsync only cares about hardlinks to regular files. I - guess you could also use them for sockets, devices and other beasts, - but I have not seen them. +rsyncd over ssh - When trying to reproduce hard links, we only need to worry about - files that have more than one name (nlinks>1 && !S_ISDIR). + There are already some patches to do this. - The basic point of this is to discover alternate names that refer to - the same file. All operations, including creating the file and - writing modifications to it need only to be done for the first name. - For all later names, we just create the link and then leave it - alone. + BitKeeper uses a server whose login shell is set to bkd. That's + probably a reasonable approach. - If hard links are to be preserved: + -- -- - Before the generator/receiver fork, the list of files is received - from the sender (recv_file_list), and a table for detecting hard - links is built. - The generator looks for hard links within the file list and does - not send checksums for them, though it does send other metadata. +Use chroot only if supported - The sender sends the device number and inode with file entries, so - that files are uniquely identified. + If the platform doesn't support it, then don't even try. - The receiver goes through and creates hard links (do_hard_links) - after all data has been written, but before directory permissions - are set. + If running as non-root, then don't fail, just give a warning. + (There was a thread about this a while ago?) - At the moment device and inum are sent as 4-byte integers, which - will probably cause problems on large filesystems. On Linux the - kernel uses 64-bit ino_t's internally, and people will soon have - filesystems big enough to use them. We ought to follow NFS4 in - using 64-bit device and inode identification, perhaps with a - protocol version bump. + http://lists.samba.org/pipermail/rsync/2001-August/thread.html + http://lists.samba.org/pipermail/rsync/2001-September/thread.html - Once we've seen all the names for a particular file, we no longer - need to think about it and we can deallocate the memory. + -- -- - We can also have the case where there are links to a file that are - not in the tree being transferred. There's nothing we can do about - that. Because we rename the destination into place after writing, - any hardlinks to the old file are always going to be orphaned. In - fact that is almost necessary because otherwise we'd get really - confused if we were generating checksums for one name of a file and - modifying another. - At the moment the code seems to make a whole second copy of the file - list, which seems unnecessary. +Allow supplementary groups in rsyncd.conf 2002/04/09 - We should have a test case that exercises hard links. Since it - might be hard to compare ./tls output where the inodes change we - might need a little program to check whether several names refer to - the same file. + Perhaps allow supplementary groups to be specified in rsyncd.conf; + then make the first one the primary gid and all the rest be + supplementary gids. + -- -- Handling IPv6 on old machines @@ -325,6 +286,8 @@ Handling IPv6 on old machines We could drop the rather large lib/getaddrinfo files. + -- -- + Other IPv6 stuff: @@ -347,91 +310,43 @@ Other IPv6 stuff: which should just take a small change to the parser code. - -Errors - - If we hang or get SIGINT, then explain where we were up to. Perhaps - have a static buffer that contains the current function name, or - some kind of description of what we were trying to do. This is a - little easier on people than needing to run strace/truss. - - "The dungeon collapses! You are killed." Rather than "unexpected - eof" give a message that is more detailed if possible and also more - helpful. - - If we get an error writing to a socket, then we should perhaps - continue trying to read to see if an error message comes across - explaining why the socket is closed. I'm not sure if this would - work, but it would certainly make our messages more helpful. - - What happens if a directory is missing -x attributes. Do we lose - our load? (Debian #28416) Probably fixed now, but a test case would - be good. + -- -- -File attributes +Add ACL support 2001/12/02 Transfer ACLs. Need to think of a standard representation. Probably better not to even try to convert between NT and POSIX. Possibly can share some code with Samba. -Empty directories + -- -- + + +Lazy directory creation With the current common --include '*/' --exclude '*' pattern, people can end up with many empty directories. We might avoid this by lazily creating such directories. + -- -- -zlib - Perhaps don't use our own zlib. +Conditional -z for old protocols - Advantages: - - - will automatically be up to date with bugfixes in zlib + After we get the @RSYNCD greeting from the server, we know it's + version but we have not yet sent the command line, so we could just + remove the -z option if the server is too old. - - can leave it out for small rsync on e.g. recovery disks + For ssh invocation it's not so simple, because we actually use the + command line to start the remote process. However, we only actually + do compression in token.c, and we could therefore once we discover + the remote version emit an error if it's too old. I'm not sure if + that's a good tradeoff or not. - - can use a shared library - - - avoids people breaking rsync by trying to do this themselves and - messing up - - Should we ship zlib for systems that don't have it, or require - people to install it separately? - - Apparently this will make us incompatible with versions of rsync - that use the patched version of rsync. Probably the simplest way to - do this is to just disable gzip (with a warning) when talking to old - versions. - - -logging - - Perhaps flush stdout after each filename, so that people trying to - monitor progress in a log file can do so more easily. See - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 - - At the connections that just get a list of modules are not logged, - but they should be. - - If a child of the rsync daemon dies with a signal, we should notice - that when we reap it and log a message. + -- -- - Keep stderr and stdout properly separated (Debian #23626) - After we get the @RSYNCD greeting from the server, we know it's - version but we have not yet sent the command line, so we could just - remove the -z option if the server is too old. - - For ssh invocation it's not so simple, because we actually use the - command line to start the remote process. However, we only actually - do compression in token.c, and we could therefore once we discover - the remote version emit an error if it's too old. I'm not sure if - that's a good tradeoff or not. - - -proxy authentication +proxy authentication 2002/01/23 Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do HTTP Basic Proxy-Authentication. @@ -439,11 +354,17 @@ proxy authentication Multiple schemes are possible, up to and including the insanity that is NTLM, but Basic probably covers most cases. -SOCKS + -- -- + + +SOCKS 2002/01/23 Add --with-socks, and then perhaps a command-line option to put them on or off. This might be more reliable than LD_PRELOAD hacks. + -- -- + + FAT support rsync to a FAT partition on a Unix machine doesn't work very well at @@ -453,37 +374,17 @@ FAT support I guess the code to do this is currently #ifdef'd on Windows; perhaps we ought to intelligently fall back to it on Unix too. + -- -- -Better statistics: - - mbp: hey, how about an rsync option that just gives you the - summary without the list of files? And perhaps gives more - information like the number of new files, number of changed, - deleted, etc. ? Rasmus: nice idea there is --stats - but at the moment it's very tridge-oriented rather than - user-friendly it would be nice to improve it that would - also work well with --dryrun -TDB: +Allow forcing arbitrary permissions 2002/03/12 - Rather than storing the file list in memory, store it in a TDB. - - This *might* make memory usage lower while building the file list. - - Hashtable lookup will mean files are not transmitted in order, - though... hm. - - This would neatly eliminate one of the major post-fork shared data - structures. - - -chmod: - - On 12 Mar 2002, Dave Dykstra wrote: > If we -would add an option to do that functionality, I would vote for one > -that was more general which could mask off any set of permission bits -and > possibly add any set of bits. Perhaps a chmod-like syntax if it -could be > implemented simply. + On 12 Mar 2002, Dave Dykstra wrote: + > If we would add an option to do that functionality, I + > would vote for one that was more general which could mask + > off any set of permission bits and possibly add any set of + > bits. Perhaps a chmod-like syntax if it could be + > implemented simply. I think that would be good too. For example, people uploading files to a web server might like to say @@ -500,8 +401,10 @@ could be > implemented simply. (Debian #23628) + -- -- ---diff + +--diff david.e.sewell 2002/03/15 Allow people to specify the diff command. (Might want to use wdiff, gnudiff, etc.) @@ -513,63 +416,281 @@ could be > implemented simply. Security interactions with daemon mode? - (Suggestion from david.e.sewell) + -- -- -Incorrect timestamps (Debian #100295) +Add daemon --no-detach and --no-fork options - A bit hard to believe, but apparently it happens. + Very useful for debugging. Also good when running under a + daemon-monitoring process that tries to restart the service when the + parent exits. + -- -- -Check "refuse options works" +DOCUMENTATION -------------------------------------------------------- - We need a test case for this... +Update README - Was this broken when we changed to popt? + -- -- -PERFORMANCE ---------------------------------------------------------- +Keep list of open issues and todos on the web site -MD4 file_sum + -- -- - If we're doing a local transfer, or using -W, then perhaps don't - send the file checksum. If we're doing a local transfer, then - calculating MD4 checksums uses 90% of CPU and is unlikely to be - useful. - Indeed for transfers over zlib or ssh we can also rely on the - transport to have quite strong protection against corruption. +Update web site from CVS - Perhaps we should have an option to disable this, analogous to ---whole-file, although it would default to disabled. The file -checksum takes up a definite space in the protocol -- we can either -set it to 0, or perhaps just leave it out. + -- -- -MD4 - Perhaps borrow an assembler MD4 from someone? +Perhaps redo manual as SGML - Make sure we call MD4 with properly-sized blocks whenever possible - to avoid copying into the residue region? + The man page is getting rather large, and there is more information + that ought to be added. -String area code + TexInfo source is probably a dying format. - Test whether this is actually faster than just using malloc(). If - it's not (anymore), throw it out. - + Linuxdoc looks like the most likely contender. I know DocBook is + favoured by some people, but it's so bloody verbose, even with emacs + support. -PLATFORMS ------------------------------------------------------------ + -- -- -Win32 +LOGGING -------------------------------------------------------------- - Don't detach, because this messes up --srvany. +Make dry run list all updates 2002/04/03 + + --dry-run is too dry + + Mark Santcroos points out that -n fails to list files which have + only metadata changes, though it probably should. + + There may be a Debian bug about this as well. + + -- -- + + +Memory accounting + + At exit, show how much memory was used for the file list, etc. + + Also we do a wierd exponential-growth allocation in flist.c. I'm + not sure this makes sense with modern mallocs. At any rate it will + make us allocate a huge amount of memory for large file lists. + + -- -- + + +Improve error messages + + If we hang or get SIGINT, then explain where we were up to. Perhaps + have a static buffer that contains the current function name, or + some kind of description of what we were trying to do. This is a + little easier on people than needing to run strace/truss. + + "The dungeon collapses! You are killed." Rather than "unexpected + eof" give a message that is more detailed if possible and also more + helpful. + + If we get an error writing to a socket, then we should perhaps + continue trying to read to see if an error message comes across + explaining why the socket is closed. I'm not sure if this would + work, but it would certainly make our messages more helpful. + + What happens if a directory is missing -x attributes. Do we lose + our load? (Debian #28416) Probably fixed now, but a test case would + be good. - http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html -DEVELOPMENT ---------------------------------------------------------- + -- -- -Splint + +Better statistics: Rasmus 2002/03/08 + + + hey, how about an rsync option that just gives you the + summary without the list of files? And perhaps gives + more information like the number of new files, number + of changed, deleted, etc. ? + + + nice idea there is --stats but at the moment it's very + tridge-oriented rather than user-friendly it would be + nice to improve it that would also work well with + --dryrun + + -- -- + + +Perhaps flush stdout like syslog + + Perhaps flush stdout after each filename, so that people trying to + monitor progress in a log file can do so more easily. See + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 + + -- -- + + +Log deamon sessions that just list modules + + At the connections that just get a list of modules are not logged, + but they should be. + + -- -- + + +Log child death on signal + + If a child of the rsync daemon dies with a signal, we should notice + that when we reap it and log a message. + + -- -- + + +Keep stderr and stdout properly separated (Debian #23626) + + -- -- + + +Log errors with function that reports process of origin + + Use a separate function for reporting errors; prefix it with + "rsync:" or "rsync(remote)", or perhaps even "rsync(local + generator): ". + + -- -- + + +verbose output David Stein 2001/12/20 + + Indicate whether files are new, updated, or deleted + + At end of transfer, show how many files were or were not transferred + correctly. + + -- -- + + +Add reason for transfer to file logging + + Explain *why* every file is transferred or not (e.g. "local mtime + 123123 newer than 1283198") + + -- -- + + +debugging of daemon 2002/04/08 + + Add an rsyncd.conf parameter to turn on debugging on the server. + + -- -- + + +internationalization + + Change to using gettext(). Probably need to ship this for platforms + that don't have it. + + Solicit translations. + + Does anyone care? Before we bother modifying the code, we ought to + get the manual translated first, because that's possibly more useful + and at any rate demonstrates desire. + + -- -- + +DEVELOPMENT -------------------------------------------------------- + +Handling duplicate names + + We need to be careful of duplicate names getting into the file list. + See clean_flist(). This could happen if multiple arguments include + the same file. Bad. + + I think duplicates are only a problem if they're both flowing + through the pipeline at the same time. For example we might have + updated the first occurrence after reading the checksums for the + second. So possibly we just need to make sure that we don't have + both in the pipeline at the same time. + + Possibly if we did one directory at a time that would be sufficient. + + Alternatively we could pre-process the arguments to make sure no + duplicates will ever be inserted. There could be some bad cases + when we're collapsing symlinks. + + We could have a hash table. + + The root of the problem is that we do not want more than one file + list entry referring to the same file. At first glance there are + several ways this could happen: symlinks, hardlinks, and repeated + names on the command line. + + If names are repeated on the command line, they may be present in + different forms, perhaps by traversing directory paths in different + ways, traversing paths including symlinks. Also we need to allow + for expansion of globs by rsync. + + At the moment, clean_flist() requires having the entire file list in + memory. Duplicate names are detected just by a string comparison. + + We don't need to worry about hard links causing duplicates because + files are never updated in place. Similarly for symlinks. + + I think even if we're using a different symlink mode we don't need + to worry. + + Unless we're really clever this will introduce a protocol + incompatibility, so we need to be able to accept the old format as + well. + + -- -- + + +Use generic zlib 2002/02/25 + + Perhaps don't use our own zlib. + + Advantages: + + - will automatically be up to date with bugfixes in zlib + + - can leave it out for small rsync on e.g. recovery disks + + - can use a shared library + + - avoids people breaking rsync by trying to do this themselves and + messing up + + Should we ship zlib for systems that don't have it, or require + people to install it separately? + + Apparently this will make us incompatible with versions of rsync + that use the patched version of rsync. Probably the simplest way to + do this is to just disable gzip (with a warning) when talking to old + versions. + + -- -- + + +TDB: 2002/03/12 + + Rather than storing the file list in memory, store it in a TDB. + + This *might* make memory usage lower while building the file list. + + Hashtable lookup will mean files are not transmitted in order, + though... hm. + + This would neatly eliminate one of the major post-fork shared data + structures. + + -- -- + + +Splint 2002/03/12 Build rsync with SPLINT to try to find security holes. Add annotations as necessary. Keep track of the number of warnings @@ -577,14 +698,8 @@ Splint security bugs. Knowing the percentage of likely hits would be really interesting for other projects. -Torture test - - Something that just keeps running rsync continuously over a data set - likely to generate problems. + -- -- -Cross-testing - - Run current rsync versions against significant past releases. Memory debugger @@ -592,168 +707,276 @@ Memory debugger http://devel-home.kde.org/~sewardj/ -Release script + -- -- + + +Create release script - Update spec files + Script would: - Build tar file; upload + Update spec files - Send announcement to mailing list and c.o.l.a. + Build tar file; upload + + Send announcement to mailing list and c.o.l.a. - Make freshmeat announcement + Make freshmeat announcement - Update web site + Update web site + -- -- -TESTING -------------------------------------------------------------- +Add machines to build farm -Cross-test versions + Cygwin (on different versions of Win32?) - Part of the regression suite should be making sure that we don't - break backwards compatibility: old clients vs new servers and so on. - Ideally we would test both up and down from the current release to - all old versions. + HP-UX variants (via HP?) - We might need to omit broken old versions, or versions in which - particular functionality is broken + SCO - It might be sufficient to test downloads from well-known public - rsync servers running different versions of rsync. This will give - some testing and also be the most common case for having different - versions and not being able to upgrade. -Test on kernel source + -- -- - Download all versions of kernel; unpack, sync between them. Also - sync between uncompressed tarballs. Compare directories after - transfer. +PERFORMANCE ---------------------------------------------------------- - Use local mode; ssh; daemon; --whole-file and --no-whole-file. +File list structure in memory - Use awk to pull out the 'speedup' number for each transfer. Make - sure it is >= x. + Rather than one big array, perhaps have a tree in memory mirroring + the directory tree. + This might make sorting much faster! (I'm not sure it's a big CPU + problem, mind you.) -Test large files + It might also reduce memory use in storing repeated directory names + -- again I'm not sure this is a problem. - Sparse and non-sparse + -- -- -Mutator program - Insert bytes, delete bytes, swap blocks, ... +Traverse just one directory at a time -configure option to enable dangerous tests + Traverse just one directory at a time. Tridge says it's possible. -If tests are skipped, say why. + At the moment rsync reads the whole file list into memory at the + start, which makes us use a lot of memory and also not pipeline + network access as much as we could. -Test daemon feature to disallow particular options. + -- -- -Pipe program that makes slow/jerky connections. -Versions of read() and write() that corrupt the stream, or abruptly -fail +Hard-link handling -Separate makefile target to run rough tests -- or perhaps just run -them every time? + At the moment hardlink handling is very expensive, so it's off by + default. It does not need to be so. -Test "refuse options" works + Since most of the solutions are rather intertwined with the file + list it is probably better to fix that first, although fixing + hardlinks is possibly simpler. - What about for --recursive? + We can rule out hardlinked directories since they will probably + screw us up in all kinds of ways. They simply should not be used. - If you specify an unrecognized option here, you should get an error. + At the moment rsync only cares about hardlinks to regular files. I + guess you could also use them for sockets, devices and other beasts, + but I have not seen them. + When trying to reproduce hard links, we only need to worry about + files that have more than one name (nlinks>1 && !S_ISDIR). -DOCUMENTATION -------------------------------------------------------- + The basic point of this is to discover alternate names that refer to + the same file. All operations, including creating the file and + writing modifications to it need only to be done for the first name. + For all later names, we just create the link and then leave it + alone. -Update README + If hard links are to be preserved: -Keep list of open issues and todos on the web site + Before the generator/receiver fork, the list of files is received + from the sender (recv_file_list), and a table for detecting hard + links is built. -Update web site from CVS + The generator looks for hard links within the file list and does + not send checksums for them, though it does send other metadata. + The sender sends the device number and inode with file entries, so + that files are uniquely identified. -Perhaps redo manual as SGML + The receiver goes through and creates hard links (do_hard_links) + after all data has been written, but before directory permissions + are set. - The man page is getting rather large, and there is more information - that ought to be added. + At the moment device and inum are sent as 4-byte integers, which + will probably cause problems on large filesystems. On Linux the + kernel uses 64-bit ino_t's internally, and people will soon have + filesystems big enough to use them. We ought to follow NFS4 in + using 64-bit device and inode identification, perhaps with a + protocol version bump. - TexInfo source is probably a dying format. + Once we've seen all the names for a particular file, we no longer + need to think about it and we can deallocate the memory. - Linuxdoc looks like the most likely contender. I know DocBook is - favoured by some people, but it's so bloody verbose, even with emacs - support. + We can also have the case where there are links to a file that are + not in the tree being transferred. There's nothing we can do about + that. Because we rename the destination into place after writing, + any hardlinks to the old file are always going to be orphaned. In + fact that is almost necessary because otherwise we'd get really + confused if we were generating checksums for one name of a file and + modifying another. + At the moment the code seems to make a whole second copy of the file + list, which seems unnecessary. -BUILD FARM ----------------------------------------------------------- + We should have a test case that exercises hard links. Since it + might be hard to compare ./tls output where the inodes change we + might need a little program to check whether several names refer to + the same file. -Add machines + -- -- - Cygwin (on different versions of Win32?) - HP-UX variants (via HP?) +Allow skipping MD4 file_sum 2002/04/08 - SCO + If we're doing a local transfer, or using -W, then perhaps don't + send the file checksum. If we're doing a local transfer, then + calculating MD4 checksums uses 90% of CPU and is unlikely to be + useful. + Indeed for transfers over zlib or ssh we can also rely on the + transport to have quite strong protection against corruption. -LOGGING -------------------------------------------------------------- + Perhaps we should have an option to disable this, + analogous to --whole-file, although it would default to + disabled. The file checksum takes up a definite space in + the protocol -- we can either set it to 0, or perhaps just + leave it out. - Perhaps flush stdout after each filename, so that people trying to - monitor progress in a log file can do so more easily. See - http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 + -- -- - At the connections that just get a list of modules are not logged, - but they should be. - If a child of the rsync daemon dies with a signal, we should notice - that when we reap it and log a message. +Accelerate MD4 - Keep stderr and stdout properly separated (Debian #23626) + Perhaps borrow an assembler MD4 from someone? - Use a separate function for reporting errors; prefix it with - "rsync:" or "rsync(remote)", or perhaps even "rsync(local - generator): ". + Make sure we call MD4 with properly-sized blocks whenever possible + to avoid copying into the residue region? -verbose output - - Indicate whether files are new, updated, or deleted + -- -- - At end of transfer, show how many files were or were not transferred - correctly. --vv +String area code - Explain *why* every file is transferred or not (e.g. "local mtime - 123123 newer than 1283198") + Test whether this is actually faster than just using malloc(). If + it's not (anymore), throw it out. + -- -- -debugging of daemon +TESTING -------------------------------------------------------------- - Add an rsyncd.conf parameter to turn on debugging on the server. +Torture test + Something that just keeps running rsync continuously over a data set + likely to generate problems. + -- -- -NICE ----------------------------------------------------------------- ---no-detach and --no-fork options +Cross-test versions 2001/08/22 - Very useful for debugging. Also good when running under a - daemon-monitoring process that tries to restart the service when the - parent exits. + Part of the regression suite should be making sure that we + don't break backwards compatibility: old clients vs new + servers and so on. Ideally we would test both up and down + from the current release to all old versions. -hang/timeout friendliness + Run current rsync versions against significant past releases. -internationalization + We might need to omit broken old versions, or versions in which + particular functionality is broken - Change to using gettext(). Probably need to ship this for platforms - that don't have it. + It might be sufficient to test downloads from well-known public + rsync servers running different versions of rsync. This will give + some testing and also be the most common case for having different + versions and not being able to upgrade. - Solicit translations. + The new --protocol option may help in this. - Does anyone care? Before we bother modifying the code, we ought to - get the manual translated first, because that's possibly more useful - and at any rate demonstrates desire. + -- -- + + +Test on kernel source + + Download all versions of kernel; unpack, sync between them. Also + sync between uncompressed tarballs. Compare directories after + transfer. + + Use local mode; ssh; daemon; --whole-file and --no-whole-file. + + Use awk to pull out the 'speedup' number for each transfer. Make + sure it is >= x. + + -- -- + + +Test large files + + Sparse and non-sparse + + -- -- + + +Create mutator program for testing + + Insert bytes, delete bytes, swap blocks, ... + + -- -- + + +Create configure option to enable dangerous tests + + -- -- + + +If tests are skipped, say why. + + -- -- + + +Test daemon feature to disallow particular options. + + -- -- + + +Create pipe program for testing + + Create pipe program that makes slow/jerky connections for + testing Versions of read() and write() that corrupt the + stream, or abruptly fail + + -- -- + + +Create test makefile target for some tests + + Separate makefile target to run rough tests -- or perhaps + just run them every time? + + -- -- + + +Test "refuse options" works + + What about for --recursive? + + If you specify an unrecognized option here, you should get an error. + + We need a test case for this... + + Was this broken when we changed to popt? + + -- -- + +RELATED PROJECTS ----------------------------------------------------- rsyncsh @@ -763,22 +986,33 @@ rsyncsh current host, directory and so on. We can probably even do completion of remote filenames. + -- -- -RELATED PROJECTS ----------------------------------------------------- http://rsync.samba.org/rsync-and-debian/ + + -- -- + + rsyncable gzip patch Exhaustive, tortuous testing Cleanups? + -- -- + + rsyncsplit as alternative to real integration with gzip? + -- -- + + reverse rsync over HTTP Range Goswin Brederlow suggested this on Debian; I think tridge and I talked about it previous in relation to rproxy. + -- -- -- 2.34.1