3 BUGS ---------------------------------------------------------------
5 rsync-url barfs on upload
7 rsync foo rsync://localhost/transfer/
12 There seems to be a bug with hardlinks
14 mbp/2 build$ ls -l /tmp/a /tmp/b -i
17 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
18 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
19 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
20 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
21 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
22 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
23 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
24 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
28 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
29 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
30 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
31 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
32 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
33 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
34 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
35 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
36 mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
37 building file list ... done
38 created directory /tmp/b
44 wrote 350 bytes read 52 bytes 804.00 bytes/sec
45 total size is 232 speedup is 0.58
46 mbp/2 build$ rm -r /tmp/b
47 mbp/2 build$ ls -l /tmp/b
48 ls: /tmp/b: No such file or directory
49 mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
50 rm: cannot remove `/tmp/b': No such file or directory
51 mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
52 building file list ... done
53 created directory /tmp/b
59 wrote 350 bytes read 52 bytes 804.00 bytes/sec
60 total size is 232 speedup is 0.58
61 mbp/2 build$ ls -l /tmp/b
63 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
64 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
65 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
66 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
67 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
68 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
69 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
70 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
71 mbp/2 build$ ls -l /tmp/a
73 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
74 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
75 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
76 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
77 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
78 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
79 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
80 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
83 Progress indicator can produce corrupt output when transferring directories:
86 main/binary-arm/admin/
88 main/binary-arm/comm/8.56kB/s 0:00:52
89 main/binary-arm/devel/
91 main/binary-arm/editors/
92 main/binary-arm/electronics/s 0:00:53
93 main/binary-arm/games/
94 main/binary-arm/graphics/
95 main/binary-arm/hamradio/
96 main/binary-arm/interpreters/
97 main/binary-arm/libs/6.61kB/s 0:00:54
100 main/binary-arm/misc/
103 I don't think we handle this properly on systems that don't have the
107 Part of the regression suite should be making sure that we don't
108 break backwards compatibility: old clients vs new servers and so
109 on. Ideally we would test the cross product of versions.
111 It might be sufficient to test downloads from well-known public
112 rsync servers running different versions of rsync. This will give
113 some testing and also be the most common case for having different
114 versions and not being able to upgrade.
116 --no-blocking-io might be broken
118 in the same way as --no-whole-file; somebody needs to check.
120 Do not rely on having a group called "nobody"
122 http://www.linuxbase.org/spec/refspecs/LSB_1.1.0/gLSB/usernames.html
124 On Debian it's "nogroup"
126 DAEMON --------------------------------------------------------------
128 server-imposed bandwidth limits
132 There are already some patches to do this.
134 BitKeeper uses a server whose login shell is set to bkd. That's
135 probably a reasonable approach.
138 FEATURES ------------------------------------------------------------
141 --dry-run is insufficiently dry
143 Mark Santcroos points out that -n fails to list files which have
144 only metadata changes, though it probably should.
146 There may be a Debian bug about this as well.
151 If the platform doesn't support it, then don't even try.
153 If running as non-root, then don't fail, just give a warning.
154 (There was a thread about this a while ago?)
156 http://lists.samba.org/pipermail/rsync/2001-August/thread.html
157 http://lists.samba.org/pipermail/rsync/2001-September/thread.html
162 Avoids traversal. Better option than a pile of --include statements
163 for people who want to generate the file list using a find(1)
169 Perhaps allow supplementary groups to be specified in rsyncd.conf;
170 then make the first one the primary gid and all the rest be
174 File list structure in memory
176 Rather than one big array, perhaps have a tree in memory mirroring
179 This might make sorting much faster! (I'm not sure it's a big CPU
182 It might also reduce memory use in storing repeated directory names
183 -- again I'm not sure this is a problem.
187 Traverse just one directory at a time. Tridge says it's possible.
189 At the moment rsync reads the whole file list into memory at the
190 start, which makes us use a lot of memory and also not pipeline
191 network access as much as we could.
194 Handling duplicate names
196 We need to be careful of duplicate names getting into the file list.
197 See clean_flist(). This could happen if multiple arguments include
200 I think duplicates are only a problem if they're both flowing
201 through the pipeline at the same time. For example we might have
202 updated the first occurrence after reading the checksums for the
203 second. So possibly we just need to make sure that we don't have
204 both in the pipeline at the same time.
206 Possibly if we did one directory at a time that would be sufficient.
208 Alternatively we could pre-process the arguments to make sure no
209 duplicates will ever be inserted. There could be some bad cases
210 when we're collapsing symlinks.
212 We could have a hash table.
214 The root of the problem is that we do not want more than one file
215 list entry referring to the same file. At first glance there are
216 several ways this could happen: symlinks, hardlinks, and repeated
217 names on the command line.
219 If names are repeated on the command line, they may be present in
220 different forms, perhaps by traversing directory paths in different
221 ways, traversing paths including symlinks. Also we need to allow
222 for expansion of globs by rsync.
224 At the moment, clean_flist() requires having the entire file list in
225 memory. Duplicate names are detected just by a string comparison.
227 We don't need to worry about hard links causing duplicates because
228 files are never updated in place. Similarly for symlinks.
230 I think even if we're using a different symlink mode we don't need
233 Unless we're really clever this will introduce a protocol
234 incompatibility, so we need to be able to accept the old format as
240 At exit, show how much memory was used for the file list, etc.
242 Also we do a wierd exponential-growth allocation in flist.c. I'm
243 not sure this makes sense with modern mallocs. At any rate it will
244 make us allocate a huge amount of memory for large file lists.
249 At the moment hardlink handling is very expensive, so it's off by
250 default. It does not need to be so.
252 Since most of the solutions are rather intertwined with the file
253 list it is probably better to fix that first, although fixing
254 hardlinks is possibly simpler.
256 We can rule out hardlinked directories since they will probably
257 screw us up in all kinds of ways. They simply should not be used.
259 At the moment rsync only cares about hardlinks to regular files. I
260 guess you could also use them for sockets, devices and other beasts,
261 but I have not seen them.
263 When trying to reproduce hard links, we only need to worry about
264 files that have more than one name (nlinks>1 && !S_ISDIR).
266 The basic point of this is to discover alternate names that refer to
267 the same file. All operations, including creating the file and
268 writing modifications to it need only to be done for the first name.
269 For all later names, we just create the link and then leave it
272 If hard links are to be preserved:
274 Before the generator/receiver fork, the list of files is received
275 from the sender (recv_file_list), and a table for detecting hard
278 The generator looks for hard links within the file list and does
279 not send checksums for them, though it does send other metadata.
281 The sender sends the device number and inode with file entries, so
282 that files are uniquely identified.
284 The receiver goes through and creates hard links (do_hard_links)
285 after all data has been written, but before directory permissions
288 At the moment device and inum are sent as 4-byte integers, which
289 will probably cause problems on large filesystems. On Linux the
290 kernel uses 64-bit ino_t's internally, and people will soon have
291 filesystems big enough to use them. We ought to follow NFS4 in
292 using 64-bit device and inode identification, perhaps with a
293 protocol version bump.
295 Once we've seen all the names for a particular file, we no longer
296 need to think about it and we can deallocate the memory.
298 We can also have the case where there are links to a file that are
299 not in the tree being transferred. There's nothing we can do about
300 that. Because we rename the destination into place after writing,
301 any hardlinks to the old file are always going to be orphaned. In
302 fact that is almost necessary because otherwise we'd get really
303 confused if we were generating checksums for one name of a file and
306 At the moment the code seems to make a whole second copy of the file
307 list, which seems unnecessary.
309 We should have a test case that exercises hard links. Since it
310 might be hard to compare ./tls output where the inodes change we
311 might need a little program to check whether several names refer to
316 Put back the old socket code; if on a machine that does not properly
317 support the getaddrinfo API, then use it. This is probably much
318 simpler than reimplementing it. This might get us working again on
319 RedHat 5 and similar systems. Although the Kame patch seems like a
320 good idea, in fact it is a much broader interface than the
321 relatively narrow "open by name", "accept and log" interface that
322 rsync uses internally, and it has the disadvantage of clashing with
323 half-arsed implementations of the API.
325 Implement suggestions from http://www.kame.net/newsletter/19980604/
326 and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
328 If a host has multiple addresses, then listen try to connect to all
329 in order until we get through. (getaddrinfo may return multiple
330 addresses.) This is kind of implemented already.
332 Possibly also when starting as a server we may need to listen on
333 multiple passive addresses. This might be a bit harder, because we
334 may need to select on all of them. Hm.
336 Define a syntax for IPv6 literal addresses. Since they include
337 colons, they tend to break most naming systems, including ours.
338 Based on the HTTP IPv6 syntax, I think we should use
340 rsync://[::1]/foo/bar
343 which should just take a small change to the parser code.
348 If we hang or get SIGINT, then explain where we were up to. Perhaps
349 have a static buffer that contains the current function name, or
350 some kind of description of what we were trying to do. This is a
351 little easier on people than needing to run strace/truss.
353 "The dungeon collapses! You are killed." Rather than "unexpected
354 eof" give a message that is more detailed if possible and also more
357 If we get an error writing to a socket, then we should perhaps
358 continue trying to read to see if an error message comes across
359 explaining why the socket is closed. I'm not sure if this would
360 work, but it would certainly make our messages more helpful.
362 What happens if a directory is missing -x attributes. Do we lose
363 our load? (Debian #28416) Probably fixed now, but a test case
369 Device major/minor numbers should be at least 32 bits each. See
370 http://lists.samba.org/pipermail/rsync/2001-November/005357.html
372 Transfer ACLs. Need to think of a standard representation.
373 Probably better not to even try to convert between NT and POSIX.
374 Possibly can share some code with Samba.
378 With the current common --include '*/' --exclude '*' pattern, people
379 can end up with many empty directories. We might avoid this by
380 lazily creating such directories.
385 Perhaps don't use our own zlib.
389 - will automatically be up to date with bugfixes in zlib
391 - can leave it out for small rsync on e.g. recovery disks
393 - can use a shared library
395 - avoids people breaking rsync by trying to do this themselves and
398 Should we ship zlib for systems that don't have it, or require
399 people to install it separately?
401 Apparently this will make us incompatible with versions of rsync
402 that use the patched version of rsync. Probably the simplest way to
403 do this is to just disable gzip (with a warning) when talking to old
409 Perhaps flush stdout after each filename, so that people trying to
410 monitor progress in a log file can do so more easily. See
411 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
413 At the connections that just get a list of modules are not logged,
416 If a child of the rsync daemon dies with a signal, we should notice
417 that when we reap it and log a message.
419 Keep stderr and stdout properly separated (Debian #23626)
421 After we get the @RSYNCD greeting from the server, we know it's
422 version but we have not yet sent the command line, so we could just
423 remove the -z option if the server is too old.
425 For ssh invocation it's not so simple, because we actually use the
426 command line to start the remote process. However, we only actually
427 do compression in token.c, and we could therefore once we discover
428 the remote version emit an error if it's too old. I'm not sure if
429 that's a good tradeoff or not.
434 There are already some patches to do this.
438 Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
439 HTTP Basic Proxy-Authentication.
441 Multiple schemes are possible, up to and including the insanity that
442 is NTLM, but Basic probably covers most cases.
446 Add --with-socks, and then perhaps a command-line option to put them
447 on or off. This might be more reliable than LD_PRELOAD hacks.
451 rsync to a FAT partition on a Unix machine doesn't work very well
452 at the moment. I think we get errors about invalid filenames and
453 perhaps also trying to do atomic renames.
455 I guess the code to do this is currently #ifdef'd on Windows; perhaps
456 we ought to intelligently fall back to it on Unix too.
461 <Rasmus> mbp: hey, how about an rsync option that just gives you the
462 summary without the list of files? And perhaps gives more
463 information like the number of new files, number of changed,
465 <mbp> Rasmus: nice idea
466 <mbp> there is --stats
467 <mbp> but at the moment it's very tridge-oriented
468 <mbp> rather than user-friendly
469 <mbp> it would be nice to improve it
470 <mbp> that would also work well with --dryrun
474 Rather than storing the file list in memory, store it in a TDB.
476 This *might* make memory usage lower while building the file list.
478 Hashtable lookup will mean files are not transmitted in order,
481 This would neatly eliminate one of the major post-fork shared data
487 On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote:
488 > If we would add an option to do that functionality, I would vote for one
489 > that was more general which could mask off any set of permission bits and
490 > possibly add any set of bits. Perhaps a chmod-like syntax if it could be
491 > implemented simply.
493 I think that would be good too. For example, people uploading files
494 to a web server might like to say
496 rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
498 Ideally the patch would implement as many of the gnu chmod semantics
499 as possible. I think the mode parser should be a separate function
500 that passes back something like (mask,set) description to the rest of
501 the program. For bonus points there would be a test case for the
504 Possibly also --chown
511 Allow people to specify the diff command. (Might want to use wdiff,
514 Just diff the temporary file with the destination file, and delete
515 the tmp file rather than moving it into place.
517 Interaction with --partial.
519 Security interactions with daemon mode?
521 (Suggestion from david.e.sewell)
524 Incorrect timestamps (Debian #100295)
526 A bit hard to believe, but apparently it happens.
529 Check "refuse options works"
531 We need a test case for this...
533 Was this broken when we changed to popt?
536 PERFORMANCE ----------------------------------------------------------
540 If we're doing a local transfer, or using -W, then perhaps don't
541 send the file checksum. If we're doing a local transfer, then
542 calculating MD4 checksums uses 90% of CPU and is unlikely to be
545 Indeed for transfers over zlib or ssh we can also rely on the
546 transport to have quite strong protection against corruption.
548 Perhaps we should have an option to disable this, analogous to
549 --whole-file, although it would default to disabled. The file
550 checksum takes up a definite space in the protocol -- we can either
551 set it to 0, or perhaps just leave it out.
555 Perhaps borrow an assembler MD4 from someone?
557 Make sure we call MD4 with properly-sized blocks whenever possible
558 to avoid copying into the residue region?
562 Test whether this is actually faster than just using malloc(). If
563 it's not (anymore), throw it out.
566 PLATFORMS ------------------------------------------------------------
570 Don't detach, because this messes up --srvany.
572 http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html
574 According to "Effective TCP/IP Programming" (??) close() on a socket
575 has incorrect behaviour on Windows -- it sends a RST packet to the
576 other side, which gives a "connection reset by peer" error. On that
577 platform we should probably do shutdown() instead. However, on Unix
578 we are correct to call close(), because shutdown() discards
582 DEVELOPMENT ----------------------------------------------------------
586 Build rsync with SPLINT to try to find security holes. Add
587 annotations as necessary. Keep track of the number of warnings
588 found initially, and see how many of them are real bugs, or real
589 security bugs. Knowing the percentage of likely hits would be
590 really interesting for other projects.
594 Something that just keeps running rsync continuously over a data set
595 likely to generate problems.
599 Run current rsync versions against significant past releases.
603 jra recommends Valgrind:
605 http://devel-home.kde.org/~sewardj/
611 Build tar file; upload
613 Send announcement to mailing list and c.o.l.a.
615 Make freshmeat announcement
621 TESTING --------------------------------------------------------------
625 Part of the regression suite should be making sure that we don't
626 break backwards compatibility: old clients vs new servers and so
627 on. Ideally we would test both up and down from the current release
630 We might need to omit broken old versions, or versions in which
631 particular functionality is broken
633 It might be sufficient to test downloads from well-known public
634 rsync servers running different versions of rsync. This will give
635 some testing and also be the most common case for having different
636 versions and not being able to upgrade.
639 Test on kernel source
641 Download all versions of kernel; unpack, sync between them. Also
642 sync between uncompressed tarballs. Compare directories after
645 Use local mode; ssh; daemon; --whole-file and --no-whole-file.
647 Use awk to pull out the 'speedup' number for each transfer. Make
653 Sparse and non-sparse
657 Insert bytes, delete bytes, swap blocks, ...
659 configure option to enable dangerous tests
661 If tests are skipped, say why.
663 Test daemon feature to disallow particular options.
665 Pipe program that makes slow/jerky connections.
667 Versions of read() and write() that corrupt the stream, or abruptly fail
669 Separate makefile target to run rough tests -- or perhaps just run
672 Test "refuse options" works
674 What about for --recursive?
676 If you specify an unrecognized option here, you should get an error.
679 DOCUMENTATION --------------------------------------------------------
683 Keep list of open issues and todos on the web site
685 Update web site from CVS
688 Perhaps redo manual as SGML
690 The man page is getting rather large, and there is more information
691 that ought to be added.
693 TexInfo source is probably a dying format.
695 Linuxdoc looks like the most likely contender. I know DocBook is
696 favoured by some people, but it's so bloody verbose, even with emacs
700 BUILD FARM -----------------------------------------------------------
704 AMDAHL UTS (Dave Dykstra)
706 Cygwin (on different versions of Win32?)
708 HP-UX variants (via HP?)
713 LOGGING --------------------------------------------------------------
715 Perhaps flush stdout after each filename, so that people trying to
716 monitor progress in a log file can do so more easily. See
717 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
719 At the connections that just get a list of modules are not logged,
722 If a child of the rsync daemon dies with a signal, we should notice
723 that when we reap it and log a message.
725 Keep stderr and stdout properly separated (Debian #23626)
727 Use a separate function for reporting errors; prefix it with
728 "rsync:" or "rsync(remote)", or perhaps even "rsync(local
733 Indicate whether files are new, updated, or deleted
735 At end of transfer, show how many files were or were not transferred
740 Explain *why* every file is transferred or not (e.g. "local mtime
741 123123 newer than 1283198")
746 Add an rsyncd.conf parameter to turn on debugging on the server.
750 NICE -----------------------------------------------------------------
752 --no-detach and --no-fork options
754 Very useful for debugging. Also good when running under a
755 daemon-monitoring process that tries to restart the service when the
758 hang/timeout friendliness
762 Change to using gettext(). Probably need to ship this for platforms
765 Solicit translations.
767 Does anyone care? Before we bother modifying the code, we ought to
768 get the manual translated first, because that's possibly more useful
769 and at any rate demonstrates desire.
773 Write a small emulation of interactive ftp as a Pythonn program
774 that calls rsync. Commands such as "cd", "ls", "ls *.c" etc map
775 fairly directly into rsync commands: it just needs to remember the
776 current host, directory and so on. We can probably even do
777 completion of remote filenames.
780 RELATED PROJECTS -----------------------------------------------------
782 http://rsync.samba.org/rsync-and-debian/
786 Exhaustive, tortuous testing
790 rsyncsplit as alternative to real integration with gzip?
792 reverse rsync over HTTP Range
794 Goswin Brederlow suggested this on Debian; I think tridge and I
795 talked about it previous in relation to rproxy.