X-Git-Url: https://mattmccutchen.net/rsync/rsync.git/blobdiff_plain/5aafd07b37b997696f81d98f4c0c54aab0f98041..b35d0d8e9ae9c5407c9f781b545f8a66b9caa9d0:/TODO diff --git a/TODO b/TODO index 88216ad6..a24ef7a9 100644 --- a/TODO +++ b/TODO @@ -1,21 +1,126 @@ -*- indented-text -*- -URGENT --------------------------------------------------------------- +BUGS --------------------------------------------------------------- + +There seems to be a bug with hardlinks + + mbp/2 build$ ls -l /tmp/a /tmp/b -i + /tmp/a: + total 32 + 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + + /tmp/b: + total 32 + 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b + building file list ... done + created directory /tmp/b + ./ + a1 + a4 + a2 => a1 + a3 => a2 + wrote 350 bytes read 52 bytes 804.00 bytes/sec + total size is 232 speedup is 0.58 + mbp/2 build$ rm -r /tmp/b + mbp/2 build$ ls -l /tmp/b + ls: /tmp/b: No such file or directory + mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b + rm: cannot remove `/tmp/b': No such file or directory + mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b + building file list ... done + created directory /tmp/b + ./ + a1 + a4 + a2 => a1 + a3 => a2 + wrote 350 bytes read 52 bytes 804.00 bytes/sec + total size is 232 speedup is 0.58 + mbp/2 build$ ls -l /tmp/b + total 32 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + mbp/2 build$ ls -l /tmp/a + total 32 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2 + -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2 + -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3 + + +Progress indicator can produce corrupt output when transferring directories: + + main/binary-arm/ + main/binary-arm/admin/ + main/binary-arm/base/ + main/binary-arm/comm/8.56kB/s 0:00:52 + main/binary-arm/devel/ + main/binary-arm/doc/ + main/binary-arm/editors/ + main/binary-arm/electronics/s 0:00:53 + main/binary-arm/games/ + main/binary-arm/graphics/ + main/binary-arm/hamradio/ + main/binary-arm/interpreters/ + main/binary-arm/libs/6.61kB/s 0:00:54 + main/binary-arm/mail/ + main/binary-arm/math/ + main/binary-arm/misc/ + +lchmod + + I don't think we handle this properly on systems that don't have the + call. + + +DAEMON -------------------------------------------------------------- + +server-imposed bandwidth limits + +rsyncd over ssh + + There are already some patches to do this. + + BitKeeper uses a server whose login shell is set to bkd. That's + probably a reasonable approach. + + +FEATURES ------------------------------------------------------------ + + +--dry-run is insufficiently dry + + Mark Santcroos points out that -n fails to list files which have + only metadata changes, though it probably should. + + There may be a Debian bug about this as well. -IMPORTANT ------------------------------------------------------------ - -Cross-test versions - - Part of the regression suite should be making sure that we don't - break backwards compatibility: old clients vs new servers and so - on. Ideally we would test the cross product of versions. - - It might be sufficient to test downloads from well-known public - rsync servers running different versions of rsync. This will give - some testing and also be the most common case for having different - versions and not being able to upgrade. - use chroot If the platform doesn't support it, then don't even try. @@ -26,21 +131,167 @@ use chroot http://lists.samba.org/pipermail/rsync/2001-August/thread.html http://lists.samba.org/pipermail/rsync/2001-September/thread.html + --files-from Avoids traversal. Better option than a pile of --include statements for people who want to generate the file list using a find(1) command or a script. + +File list structure in memory + + Rather than one big array, perhaps have a tree in memory mirroring + the directory tree. + + This might make sorting much faster! (I'm not sure it's a big CPU + problem, mind you.) + + It might also reduce memory use in storing repeated directory names + -- again I'm not sure this is a problem. + Performance Traverse just one directory at a time. Tridge says it's possible. - - Can possibly also be smarter about memory use while looking for hard - links by reducing the refcount as we find alternative names. + + At the moment rsync reads the whole file list into memory at the + start, which makes us use a lot of memory and also not pipeline + network access as much as we could. + + +Handling duplicate names + + We need to be careful of duplicate names getting into the file list. + See clean_flist(). This could happen if multiple arguments include + the same file. Bad. + + I think duplicates are only a problem if they're both flowing + through the pipeline at the same time. For example we might have + updated the first occurrence after reading the checksums for the + second. So possibly we just need to make sure that we don't have + both in the pipeline at the same time. + + Possibly if we did one directory at a time that would be sufficient. + + Alternatively we could pre-process the arguments to make sure no + duplicates will ever be inserted. There could be some bad cases + when we're collapsing symlinks. + + We could have a hash table. + + The root of the problem is that we do not want more than one file + list entry referring to the same file. At first glance there are + several ways this could happen: symlinks, hardlinks, and repeated + names on the command line. + + If names are repeated on the command line, they may be present in + different forms, perhaps by traversing directory paths in different + ways, traversing paths including symlinks. Also we need to allow + for expansion of globs by rsync. + + At the moment, clean_flist() requires having the entire file list in + memory. Duplicate names are detected just by a string comparison. + + We don't need to worry about hard links causing duplicates because + files are never updated in place. Similarly for symlinks. + + I think even if we're using a different symlink mode we don't need + to worry. + + Unless we're really clever this will introduce a protocol + incompatibility, so we need to be able to accept the old format as + well. + + +Memory accounting + + At exit, show how much memory was used for the file list, etc. + + Also we do a wierd exponential-growth allocation in flist.c. I'm + not sure this makes sense with modern mallocs. At any rate it will + make us allocate a huge amount of memory for large file lists. + + +Hard-link handling + + At the moment hardlink handling is very expensive, so it's off by + default. It does not need to be so. + + Since most of the solutions are rather intertwined with the file + list it is probably better to fix that first, although fixing + hardlinks is possibly simpler. + + We can rule out hardlinked directories since they will probably + screw us up in all kinds of ways. They simply should not be used. + + At the moment rsync only cares about hardlinks to regular files. I + guess you could also use them for sockets, devices and other beasts, + but I have not seen them. + + When trying to reproduce hard links, we only need to worry about + files that have more than one name (nlinks>1 && !S_ISDIR). + + The basic point of this is to discover alternate names that refer to + the same file. All operations, including creating the file and + writing modifications to it need only to be done for the first name. + For all later names, we just create the link and then leave it + alone. + + If hard links are to be preserved: + + Before the generator/receiver fork, the list of files is received + from the sender (recv_file_list), and a table for detecting hard + links is built. + + The generator looks for hard links within the file list and does + not send checksums for them, though it does send other metadata. + + The sender sends the device number and inode with file entries, so + that files are uniquely identified. + + The receiver goes through and creates hard links (do_hard_links) + after all data has been written, but before directory permissions + are set. + + At the moment device and inum are sent as 4-byte integers, which + will probably cause problems on large filesystems. On Linux the + kernel uses 64-bit ino_t's internally, and people will soon have + filesystems big enough to use them. We ought to follow NFS4 in + using 64-bit device and inode identification, perhaps with a + protocol version bump. + + Once we've seen all the names for a particular file, we no longer + need to think about it and we can deallocate the memory. + + We can also have the case where there are links to a file that are + not in the tree being transferred. There's nothing we can do about + that. Because we rename the destination into place after writing, + any hardlinks to the old file are always going to be orphaned. In + fact that is almost necessary because otherwise we'd get really + confused if we were generating checksums for one name of a file and + modifying another. + + At the moment the code seems to make a whole second copy of the file + list, which seems unnecessary. + + We should have a test case that exercises hard links. Since it + might be hard to compare ./tls output where the inodes change we + might need a little program to check whether several names refer to + the same file. IPv6 + Implement suggestions from http://www.kame.net/newsletter/19980604/ + and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt + + If a host has multiple addresses, then listen try to connect to all + in order until we get through. (getaddrinfo may return multiple + addresses.) This is kind of implemented already. + + Possibly also when starting as a server we may need to listen on + multiple passive addresses. This might be a bit harder, because we + may need to select on all of them. Hm. + Define a syntax for IPv6 literal addresses. Since they include colons, they tend to break most naming systems, including ours. Based on the HTTP IPv6 syntax, I think we should use @@ -50,6 +301,7 @@ IPv6 which should just take a small change to the parser code. + Errors If we hang or get SIGINT, then explain where we were up to. Perhaps @@ -61,6 +313,150 @@ Errors eof" give a message that is more detailed if possible and also more helpful. + If we get an error writing to a socket, then we should perhaps + continue trying to read to see if an error message comes across + explaining why the socket is closed. I'm not sure if this would + work, but it would certainly make our messages more helpful. + + What happens if a directory is missing -x attributes. Do we lose + our load? (Debian #28416) Probably fixed now, but a test case + would be good. + + +File attributes + + Device major/minor numbers should be at least 32 bits each. See + http://lists.samba.org/pipermail/rsync/2001-November/005357.html + + Transfer ACLs. Need to think of a standard representation. + Probably better not to even try to convert between NT and POSIX. + Possibly can share some code with Samba. + +Empty directories + + With the current common --include '*/' --exclude '*' pattern, people + can end up with many empty directories. We might avoid this by + lazily creating such directories. + + +zlib + + Perhaps don't use our own zlib. + + Advantages: + + - will automatically be up to date with bugfixes in zlib + + - can leave it out for small rsync on e.g. recovery disks + + - can use a shared library + + - avoids people breaking rsync by trying to do this themselves and + messing up + + Should we ship zlib for systems that don't have it, or require + people to install it separately? + + Apparently this will make us incompatible with versions of rsync + that use the patched version of rsync. Probably the simplest way to + do this is to just disable gzip (with a warning) when talking to old + versions. + + +proxy authentication + + Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do + HTTP Basic Proxy-Authentication. + + Multiple schemes are possible, up to and including the insanity that + is NTLM, but Basic probably covers most cases. + +SOCKS + + Add --with-socks, and then perhaps a command-line option to put them + on or off. This might be more reliable than LD_PRELOAD hacks. + +Better statistics: + + mbp: hey, how about an rsync option that just gives you the + summary without the list of files? And perhaps gives more + information like the number of new files, number of changed, + deleted, etc. ? + Rasmus: nice idea + there is --stats + but at the moment it's very tridge-oriented + rather than user-friendly + it would be nice to improve it + that would also work well with --dryrun + +TDB: + + Rather than storing the file list in memory, store it in a TDB. + + This *might* make memory usage lower while building the file list. + + Hashtable lookup will mean files are not transmitted in order, + though... hm. + + This would neatly eliminate one of the major post-fork shared data + structures. + + +chmod: + + On 12 Mar 2002, Dave Dykstra wrote: + > If we would add an option to do that functionality, I would vote for one + > that was more general which could mask off any set of permission bits and + > possibly add any set of bits. Perhaps a chmod-like syntax if it could be + > implemented simply. + + I think that would be good too. For example, people uploading files + to a web server might like to say + + rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/ + + Ideally the patch would implement as many of the gnu chmod semantics + as possible. I think the mode parser should be a separate function + that passes back something like (mask,set) description to the rest of + the program. For bonus points there would be a test case for the + parser. + + (Debian #23628) + + +--diff + + Allow people to specify the diff command. (Might want to use wdiff, + gnudiff, etc.) + + Just diff the temporary file with the destination file, and delete + the tmp file rather than moving it into place. + + Interaction with --partial. + + Security interactions with daemon mode? + + (Suggestion from david.e.sewell) + + +Incorrect timestamps (Debian #100295) + + A bit hard to believe, but apparently it happens. + + +Check "refuse options works" + + We need a test case for this... + + Was this broken when we changed to popt? + + +String area code + + Test whether this is actually faster than just using malloc(). If + it's not (anymore), throw it out. + + PLATFORMS ------------------------------------------------------------ @@ -77,6 +473,74 @@ Win32 we are correct to call close(), because shutdown() discards untransmitted data. +DEVELOPMENT ---------------------------------------------------------- + +Splint + + Build rsync with SPLINT to try to find security holes. Add + annotations as necessary. Keep track of the number of warnings + found initially, and see how many of them are real bugs, or real + security bugs. Knowing the percentage of likely hits would be + really interesting for other projects. + +Torture test + + Something that just keeps running rsync continuously over a data set + likely to generate problems. + +Cross-testing + + Run current rsync versions against significant past releases. + +Memory debugger + + jra recommends Valgrind: + + http://devel-home.kde.org/~sewardj/ + +TESTING -------------------------------------------------------------- + +Cross-test versions + + Part of the regression suite should be making sure that we don't + break backwards compatibility: old clients vs new servers and so + on. Ideally we would test the cross product of versions. + + It might be sufficient to test downloads from well-known public + rsync servers running different versions of rsync. This will give + some testing and also be the most common case for having different + versions and not being able to upgrade. + +Test large files + + Sparse and non-sparse + +Mutator program + + Insert bytes, delete bytes, swap blocks, ... + +configure option to enable dangerous tests + +If tests are skipped, say why. + +Test daemon feature to disallow particular options. + +Pipe program that makes slow/jerky connections. + +Versions of read() and write() that corrupt the stream, or abruptly fail + +Separate makefile target to run rough tests -- or perhaps just run +them every time? + + +DOCUMENTATION -------------------------------------------------------- + +Update README + +Keep list of open issues and todos on the web site + +Update web site from CVS + BUILD FARM ----------------------------------------------------------- Add machines @@ -89,6 +553,44 @@ Add machines SCO + +LOGGING -------------------------------------------------------------- + + Perhaps flush stdout after each filename, so that people trying to + monitor progress in a log file can do so more easily. See + http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108 + + At the connections that just get a list of modules are not logged, + but they should be. + + If a child of the rsync daemon dies with a signal, we should notice + that when we reap it and log a message. + + Keep stderr and stdout properly separated (Debian #23626) + + Use a separate function for reporting errors; prefix it with + "rsync:" or "rsync(remote)", or perhaps even "rsync(local + generator): ". + +verbose output + + Indicate whether files are new, updated, or deleted + + At end of transfer, show how many files were or were not transferred + correctly. + +-vv + + Explain *why* every file is transferred or not (e.g. "local mtime + 123123 newer than 1283198") + + +debugging of daemon + + Add an rsyncd.conf parameter to turn on debugging on the server. + + + NICE ----------------------------------------------------------------- --no-detach and --no-fork options @@ -99,8 +601,6 @@ NICE ----------------------------------------------------------------- hang/timeout friendliness - On - internationalization Change to using gettext(). Probably need to ship this for platforms @@ -108,7 +608,10 @@ internationalization Solicit translations. - Does anyone care? + Does anyone care? Before we bother modifying the code, we ought to + get the manual translated first, because that's possibly more useful + and at any rate demonstrates desire. + rsyncsh @@ -117,4 +620,3 @@ rsyncsh fairly directly into rsync commands: it just needs to remember the current host, directory and so on. We can probably even do completion of remote filenames. -