-*- indented-text -*-
-URGENT ---------------------------------------------------------------
+BUGS ---------------------------------------------------------------
+
+There seems to be a bug with hardlinks
+
+ mbp/2 build$ ls -l /tmp/a /tmp/b -i
+ /tmp/a:
+ total 32
+ 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
+ 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
+ 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
+ 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
+ 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
+ 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
+ 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
+ 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
+
+ /tmp/b:
+ total 32
+ 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
+ 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
+ 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
+ 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
+ 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
+ 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
+ 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
+ 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
+ mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
+ building file list ... done
+ created directory /tmp/b
+ ./
+ a1
+ a4
+ a2 => a1
+ a3 => a2
+ wrote 350 bytes read 52 bytes 804.00 bytes/sec
+ total size is 232 speedup is 0.58
+ mbp/2 build$ rm -r /tmp/b
+ mbp/2 build$ ls -l /tmp/b
+ ls: /tmp/b: No such file or directory
+ mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
+ rm: cannot remove `/tmp/b': No such file or directory
+ mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
+ building file list ... done
+ created directory /tmp/b
+ ./
+ a1
+ a4
+ a2 => a1
+ a3 => a2
+ wrote 350 bytes read 52 bytes 804.00 bytes/sec
+ total size is 232 speedup is 0.58
+ mbp/2 build$ ls -l /tmp/b
+ total 32
+ -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
+ -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
+ -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
+ mbp/2 build$ ls -l /tmp/a
+ total 32
+ -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
+ -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
+ -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
+ -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
+
+
+Progress indicator can produce corrupt output when transferring directories:
+
+ main/binary-arm/
+ main/binary-arm/admin/
+ main/binary-arm/base/
+ main/binary-arm/comm/8.56kB/s 0:00:52
+ main/binary-arm/devel/
+ main/binary-arm/doc/
+ main/binary-arm/editors/
+ main/binary-arm/electronics/s 0:00:53
+ main/binary-arm/games/
+ main/binary-arm/graphics/
+ main/binary-arm/hamradio/
+ main/binary-arm/interpreters/
+ main/binary-arm/libs/6.61kB/s 0:00:54
+ main/binary-arm/mail/
+ main/binary-arm/math/
+ main/binary-arm/misc/
+
+lchmod
+
+ I don't think we handle this properly on systems that don't have the
+ call.
+
+
+DAEMON --------------------------------------------------------------
+
+server-imposed bandwidth limits
+
+rsyncd over ssh
+
+ There are already some patches to do this.
+
+ BitKeeper uses a server whose login shell is set to bkd. That's
+ probably a reasonable approach.
+
+
+FEATURES ------------------------------------------------------------
+
+
+--dry-run is insufficiently dry
+
+ Mark Santcroos points out that -n fails to list files which have
+ only metadata changes, though it probably should.
+
+ There may be a Debian bug about this as well.
-IMPORTANT ------------------------------------------------------------
-
-Cross-test versions
-
- Part of the regression suite should be making sure that we don't
- break backwards compatibility: old clients vs new servers and so
- on. Ideally we would test the cross product of versions.
-
- It might be sufficient to test downloads from well-known public
- rsync servers running different versions of rsync. This will give
- some testing and also be the most common case for having different
- versions and not being able to upgrade.
-
use chroot
If the platform doesn't support it, then don't even try.
http://lists.samba.org/pipermail/rsync/2001-August/thread.html
http://lists.samba.org/pipermail/rsync/2001-September/thread.html
+
--files-from
Avoids traversal. Better option than a pile of --include statements
for people who want to generate the file list using a find(1)
command or a script.
+
+File list structure in memory
+
+ Rather than one big array, perhaps have a tree in memory mirroring
+ the directory tree.
+
+ This might make sorting much faster! (I'm not sure it's a big CPU
+ problem, mind you.)
+
+ It might also reduce memory use in storing repeated directory names
+ -- again I'm not sure this is a problem.
+
Performance
Traverse just one directory at a time. Tridge says it's possible.
-
- Can possibly also be smarter about memory use while looking for hard
- links by reducing the refcount as we find alternative names.
+
+ At the moment rsync reads the whole file list into memory at the
+ start, which makes us use a lot of memory and also not pipeline
+ network access as much as we could.
+
+
+Handling duplicate names
+
+ We need to be careful of duplicate names getting into the file list.
+ See clean_flist(). This could happen if multiple arguments include
+ the same file. Bad.
+
+ I think duplicates are only a problem if they're both flowing
+ through the pipeline at the same time. For example we might have
+ updated the first occurrence after reading the checksums for the
+ second. So possibly we just need to make sure that we don't have
+ both in the pipeline at the same time.
+
+ Possibly if we did one directory at a time that would be sufficient.
+
+ Alternatively we could pre-process the arguments to make sure no
+ duplicates will ever be inserted. There could be some bad cases
+ when we're collapsing symlinks.
+
+ We could have a hash table.
+
+ The root of the problem is that we do not want more than one file
+ list entry referring to the same file. At first glance there are
+ several ways this could happen: symlinks, hardlinks, and repeated
+ names on the command line.
+
+ If names are repeated on the command line, they may be present in
+ different forms, perhaps by traversing directory paths in different
+ ways, traversing paths including symlinks. Also we need to allow
+ for expansion of globs by rsync.
+
+ At the moment, clean_flist() requires having the entire file list in
+ memory. Duplicate names are detected just by a string comparison.
+
+ We don't need to worry about hard links causing duplicates because
+ files are never updated in place. Similarly for symlinks.
+
+ I think even if we're using a different symlink mode we don't need
+ to worry.
+
+ Unless we're really clever this will introduce a protocol
+ incompatibility, so we need to be able to accept the old format as
+ well.
+
+
+Memory accounting
+
+ At exit, show how much memory was used for the file list, etc.
+
+ Also we do a wierd exponential-growth allocation in flist.c. I'm
+ not sure this makes sense with modern mallocs. At any rate it will
+ make us allocate a huge amount of memory for large file lists.
+
+
+Hard-link handling
+
+ At the moment hardlink handling is very expensive, so it's off by
+ default. It does not need to be so.
+
+ Since most of the solutions are rather intertwined with the file
+ list it is probably better to fix that first, although fixing
+ hardlinks is possibly simpler.
+
+ We can rule out hardlinked directories since they will probably
+ screw us up in all kinds of ways. They simply should not be used.
+
+ At the moment rsync only cares about hardlinks to regular files. I
+ guess you could also use them for sockets, devices and other beasts,
+ but I have not seen them.
+
+ When trying to reproduce hard links, we only need to worry about
+ files that have more than one name (nlinks>1 && !S_ISDIR).
+
+ The basic point of this is to discover alternate names that refer to
+ the same file. All operations, including creating the file and
+ writing modifications to it need only to be done for the first name.
+ For all later names, we just create the link and then leave it
+ alone.
+
+ If hard links are to be preserved:
+
+ Before the generator/receiver fork, the list of files is received
+ from the sender (recv_file_list), and a table for detecting hard
+ links is built.
+
+ The generator looks for hard links within the file list and does
+ not send checksums for them, though it does send other metadata.
+
+ The sender sends the device number and inode with file entries, so
+ that files are uniquely identified.
+
+ The receiver goes through and creates hard links (do_hard_links)
+ after all data has been written, but before directory permissions
+ are set.
+
+ At the moment device and inum are sent as 4-byte integers, which
+ will probably cause problems on large filesystems. On Linux the
+ kernel uses 64-bit ino_t's internally, and people will soon have
+ filesystems big enough to use them. We ought to follow NFS4 in
+ using 64-bit device and inode identification, perhaps with a
+ protocol version bump.
+
+ Once we've seen all the names for a particular file, we no longer
+ need to think about it and we can deallocate the memory.
+
+ We can also have the case where there are links to a file that are
+ not in the tree being transferred. There's nothing we can do about
+ that. Because we rename the destination into place after writing,
+ any hardlinks to the old file are always going to be orphaned. In
+ fact that is almost necessary because otherwise we'd get really
+ confused if we were generating checksums for one name of a file and
+ modifying another.
+
+ At the moment the code seems to make a whole second copy of the file
+ list, which seems unnecessary.
+
+ We should have a test case that exercises hard links. Since it
+ might be hard to compare ./tls output where the inodes change we
+ might need a little program to check whether several names refer to
+ the same file.
IPv6
+ Implement suggestions from http://www.kame.net/newsletter/19980604/
+ and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
+
+ If a host has multiple addresses, then listen try to connect to all
+ in order until we get through. (getaddrinfo may return multiple
+ addresses.) This is kind of implemented already.
+
+ Possibly also when starting as a server we may need to listen on
+ multiple passive addresses. This might be a bit harder, because we
+ may need to select on all of them. Hm.
+
Define a syntax for IPv6 literal addresses. Since they include
colons, they tend to break most naming systems, including ours.
Based on the HTTP IPv6 syntax, I think we should use
which should just take a small change to the parser code.
+
Errors
If we hang or get SIGINT, then explain where we were up to. Perhaps
eof" give a message that is more detailed if possible and also more
helpful.
+ If we get an error writing to a socket, then we should perhaps
+ continue trying to read to see if an error message comes across
+ explaining why the socket is closed. I'm not sure if this would
+ work, but it would certainly make our messages more helpful.
+
+ What happens if a directory is missing -x attributes. Do we lose
+ our load? (Debian #28416) Probably fixed now, but a test case
+ would be good.
+
+
+File attributes
+
+ Device major/minor numbers should be at least 32 bits each. See
+ http://lists.samba.org/pipermail/rsync/2001-November/005357.html
+
+ Transfer ACLs. Need to think of a standard representation.
+ Probably better not to even try to convert between NT and POSIX.
+ Possibly can share some code with Samba.
+
+Empty directories
+
+ With the current common --include '*/' --exclude '*' pattern, people
+ can end up with many empty directories. We might avoid this by
+ lazily creating such directories.
+
+
+zlib
+
+ Perhaps don't use our own zlib.
+
+ Advantages:
+
+ - will automatically be up to date with bugfixes in zlib
+
+ - can leave it out for small rsync on e.g. recovery disks
+
+ - can use a shared library
+
+ - avoids people breaking rsync by trying to do this themselves and
+ messing up
+
+ Should we ship zlib for systems that don't have it, or require
+ people to install it separately?
+
+ Apparently this will make us incompatible with versions of rsync
+ that use the patched version of rsync. Probably the simplest way to
+ do this is to just disable gzip (with a warning) when talking to old
+ versions.
+
+
+proxy authentication
+
+ Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
+ HTTP Basic Proxy-Authentication.
+
+ Multiple schemes are possible, up to and including the insanity that
+ is NTLM, but Basic probably covers most cases.
+
+SOCKS
+
+ Add --with-socks, and then perhaps a command-line option to put them
+ on or off. This might be more reliable than LD_PRELOAD hacks.
+
+Better statistics:
+
+ <Rasmus> mbp: hey, how about an rsync option that just gives you the
+ summary without the list of files? And perhaps gives more
+ information like the number of new files, number of changed,
+ deleted, etc. ?
+ <mbp> Rasmus: nice idea
+ <mbp> there is --stats
+ <mbp> but at the moment it's very tridge-oriented
+ <mbp> rather than user-friendly
+ <mbp> it would be nice to improve it
+ <mbp> that would also work well with --dryrun
+
+TDB:
+
+ Rather than storing the file list in memory, store it in a TDB.
+
+ This *might* make memory usage lower while building the file list.
+
+ Hashtable lookup will mean files are not transmitted in order,
+ though... hm.
+
+ This would neatly eliminate one of the major post-fork shared data
+ structures.
+
+
+chmod:
+
+ On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote:
+ > If we would add an option to do that functionality, I would vote for one
+ > that was more general which could mask off any set of permission bits and
+ > possibly add any set of bits. Perhaps a chmod-like syntax if it could be
+ > implemented simply.
+
+ I think that would be good too. For example, people uploading files
+ to a web server might like to say
+
+ rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
+
+ Ideally the patch would implement as many of the gnu chmod semantics
+ as possible. I think the mode parser should be a separate function
+ that passes back something like (mask,set) description to the rest of
+ the program. For bonus points there would be a test case for the
+ parser.
+
+ (Debian #23628)
+
+
+--diff
+
+ Allow people to specify the diff command. (Might want to use wdiff,
+ gnudiff, etc.)
+
+ Just diff the temporary file with the destination file, and delete
+ the tmp file rather than moving it into place.
+
+ Interaction with --partial.
+
+ Security interactions with daemon mode?
+
+ (Suggestion from david.e.sewell)
+
+
+Incorrect timestamps (Debian #100295)
+
+ A bit hard to believe, but apparently it happens.
+
+
+Check "refuse options works"
+
+ We need a test case for this...
+
+ Was this broken when we changed to popt?
+
+
+String area code
+
+ Test whether this is actually faster than just using malloc(). If
+ it's not (anymore), throw it out.
+
+
PLATFORMS ------------------------------------------------------------
we are correct to call close(), because shutdown() discards
untransmitted data.
+DEVELOPMENT ----------------------------------------------------------
+
+Splint
+
+ Build rsync with SPLINT to try to find security holes. Add
+ annotations as necessary. Keep track of the number of warnings
+ found initially, and see how many of them are real bugs, or real
+ security bugs. Knowing the percentage of likely hits would be
+ really interesting for other projects.
+
+Torture test
+
+ Something that just keeps running rsync continuously over a data set
+ likely to generate problems.
+
+Cross-testing
+
+ Run current rsync versions against significant past releases.
+
+Memory debugger
+
+ jra recommends Valgrind:
+
+ http://devel-home.kde.org/~sewardj/
+
+TESTING --------------------------------------------------------------
+
+Cross-test versions
+
+ Part of the regression suite should be making sure that we don't
+ break backwards compatibility: old clients vs new servers and so
+ on. Ideally we would test the cross product of versions.
+
+ It might be sufficient to test downloads from well-known public
+ rsync servers running different versions of rsync. This will give
+ some testing and also be the most common case for having different
+ versions and not being able to upgrade.
+
+Test large files
+
+ Sparse and non-sparse
+
+Mutator program
+
+ Insert bytes, delete bytes, swap blocks, ...
+
+configure option to enable dangerous tests
+
+If tests are skipped, say why.
+
+Test daemon feature to disallow particular options.
+
+Pipe program that makes slow/jerky connections.
+
+Versions of read() and write() that corrupt the stream, or abruptly fail
+
+Separate makefile target to run rough tests -- or perhaps just run
+them every time?
+
+
+DOCUMENTATION --------------------------------------------------------
+
+Update README
+
+Keep list of open issues and todos on the web site
+
+Update web site from CVS
+
BUILD FARM -----------------------------------------------------------
Add machines
SCO
+
+LOGGING --------------------------------------------------------------
+
+ Perhaps flush stdout after each filename, so that people trying to
+ monitor progress in a log file can do so more easily. See
+ http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
+
+ At the connections that just get a list of modules are not logged,
+ but they should be.
+
+ If a child of the rsync daemon dies with a signal, we should notice
+ that when we reap it and log a message.
+
+ Keep stderr and stdout properly separated (Debian #23626)
+
+ Use a separate function for reporting errors; prefix it with
+ "rsync:" or "rsync(remote)", or perhaps even "rsync(local
+ generator): ".
+
+verbose output
+
+ Indicate whether files are new, updated, or deleted
+
+ At end of transfer, show how many files were or were not transferred
+ correctly.
+
+-vv
+
+ Explain *why* every file is transferred or not (e.g. "local mtime
+ 123123 newer than 1283198")
+
+
+debugging of daemon
+
+ Add an rsyncd.conf parameter to turn on debugging on the server.
+
+
+
NICE -----------------------------------------------------------------
--no-detach and --no-fork options
hang/timeout friendliness
- On
-
internationalization
Change to using gettext(). Probably need to ship this for platforms
Solicit translations.
- Does anyone care?
+ Does anyone care? Before we bother modifying the code, we ought to
+ get the manual translated first, because that's possibly more useful
+ and at any rate demonstrates desire.
+
rsyncsh
fairly directly into rsync commands: it just needs to remember the
current host, directory and so on. We can probably even do
completion of remote filenames.
-
Matt McCutchen's Web Site / rsync / rsync.git / blobdiff