3 BUGS ---------------------------------------------------------------
5 rsync-url barfs on upload
7 rsync foo rsync://localhost/transfer/
12 There seems to be a bug with hardlinks
14 mbp/2 build$ ls -l /tmp/a /tmp/b -i
17 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
18 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
19 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
20 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
21 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
22 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
23 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
24 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
28 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
29 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
30 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
31 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
32 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
33 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
34 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
35 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
36 mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
37 building file list ... done
38 created directory /tmp/b
44 wrote 350 bytes read 52 bytes 804.00 bytes/sec
45 total size is 232 speedup is 0.58
46 mbp/2 build$ rm -r /tmp/b
47 mbp/2 build$ ls -l /tmp/b
48 ls: /tmp/b: No such file or directory
49 mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
50 rm: cannot remove `/tmp/b': No such file or directory
51 mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
52 building file list ... done
53 created directory /tmp/b
59 wrote 350 bytes read 52 bytes 804.00 bytes/sec
60 total size is 232 speedup is 0.58
61 mbp/2 build$ ls -l /tmp/b
63 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
64 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
65 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
66 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
67 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
68 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
69 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
70 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
71 mbp/2 build$ ls -l /tmp/a
73 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
74 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
75 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
76 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
77 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
78 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
79 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
80 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
83 Progress indicator can produce corrupt output when transferring directories:
86 main/binary-arm/admin/
88 main/binary-arm/comm/8.56kB/s 0:00:52
89 main/binary-arm/devel/
91 main/binary-arm/editors/
92 main/binary-arm/electronics/s 0:00:53
93 main/binary-arm/games/
94 main/binary-arm/graphics/
95 main/binary-arm/hamradio/
96 main/binary-arm/interpreters/
97 main/binary-arm/libs/6.61kB/s 0:00:54
100 main/binary-arm/misc/
103 I don't think we handle this properly on systems that don't have the
107 Part of the regression suite should be making sure that we don't
108 break backwards compatibility: old clients vs new servers and so
109 on. Ideally we would test the cross product of versions.
111 It might be sufficient to test downloads from well-known public
112 rsync servers running different versions of rsync. This will give
113 some testing and also be the most common case for having different
114 versions and not being able to upgrade.
116 --no-blocking-io might be broken
118 in the same way as --no-whole-file; somebody needs to check.
120 Do not rely on having a group called "nobody"
122 http://www.linuxbase.org/spec/refspecs/LSB_1.1.0/gLSB/usernames.html
124 On Debian it's "nogroup"
126 DAEMON --------------------------------------------------------------
128 server-imposed bandwidth limits
132 There are already some patches to do this.
134 BitKeeper uses a server whose login shell is set to bkd. That's
135 probably a reasonable approach.
138 FEATURES ------------------------------------------------------------
141 --dry-run is insufficiently dry
143 Mark Santcroos points out that -n fails to list files which have
144 only metadata changes, though it probably should.
146 There may be a Debian bug about this as well.
151 If the platform doesn't support it, then don't even try.
153 If running as non-root, then don't fail, just give a warning.
154 (There was a thread about this a while ago?)
156 http://lists.samba.org/pipermail/rsync/2001-August/thread.html
157 http://lists.samba.org/pipermail/rsync/2001-September/thread.html
162 Avoids traversal. Better option than a pile of --include statements
163 for people who want to generate the file list using a find(1)
169 Perhaps allow supplementary groups to be specified in rsyncd.conf;
170 then make the first one the primary gid and all the rest be
174 File list structure in memory
176 Rather than one big array, perhaps have a tree in memory mirroring
179 This might make sorting much faster! (I'm not sure it's a big CPU
182 It might also reduce memory use in storing repeated directory names
183 -- again I'm not sure this is a problem.
187 Traverse just one directory at a time. Tridge says it's possible.
189 At the moment rsync reads the whole file list into memory at the
190 start, which makes us use a lot of memory and also not pipeline
191 network access as much as we could.
194 Handling duplicate names
196 We need to be careful of duplicate names getting into the file list.
197 See clean_flist(). This could happen if multiple arguments include
200 I think duplicates are only a problem if they're both flowing
201 through the pipeline at the same time. For example we might have
202 updated the first occurrence after reading the checksums for the
203 second. So possibly we just need to make sure that we don't have
204 both in the pipeline at the same time.
206 Possibly if we did one directory at a time that would be sufficient.
208 Alternatively we could pre-process the arguments to make sure no
209 duplicates will ever be inserted. There could be some bad cases
210 when we're collapsing symlinks.
212 We could have a hash table.
214 The root of the problem is that we do not want more than one file
215 list entry referring to the same file. At first glance there are
216 several ways this could happen: symlinks, hardlinks, and repeated
217 names on the command line.
219 If names are repeated on the command line, they may be present in
220 different forms, perhaps by traversing directory paths in different
221 ways, traversing paths including symlinks. Also we need to allow
222 for expansion of globs by rsync.
224 At the moment, clean_flist() requires having the entire file list in
225 memory. Duplicate names are detected just by a string comparison.
227 We don't need to worry about hard links causing duplicates because
228 files are never updated in place. Similarly for symlinks.
230 I think even if we're using a different symlink mode we don't need
233 Unless we're really clever this will introduce a protocol
234 incompatibility, so we need to be able to accept the old format as
240 At exit, show how much memory was used for the file list, etc.
242 Also we do a wierd exponential-growth allocation in flist.c. I'm
243 not sure this makes sense with modern mallocs. At any rate it will
244 make us allocate a huge amount of memory for large file lists.
249 At the moment hardlink handling is very expensive, so it's off by
250 default. It does not need to be so.
252 Since most of the solutions are rather intertwined with the file
253 list it is probably better to fix that first, although fixing
254 hardlinks is possibly simpler.
256 We can rule out hardlinked directories since they will probably
257 screw us up in all kinds of ways. They simply should not be used.
259 At the moment rsync only cares about hardlinks to regular files. I
260 guess you could also use them for sockets, devices and other beasts,
261 but I have not seen them.
263 When trying to reproduce hard links, we only need to worry about
264 files that have more than one name (nlinks>1 && !S_ISDIR).
266 The basic point of this is to discover alternate names that refer to
267 the same file. All operations, including creating the file and
268 writing modifications to it need only to be done for the first name.
269 For all later names, we just create the link and then leave it
272 If hard links are to be preserved:
274 Before the generator/receiver fork, the list of files is received
275 from the sender (recv_file_list), and a table for detecting hard
278 The generator looks for hard links within the file list and does
279 not send checksums for them, though it does send other metadata.
281 The sender sends the device number and inode with file entries, so
282 that files are uniquely identified.
284 The receiver goes through and creates hard links (do_hard_links)
285 after all data has been written, but before directory permissions
288 At the moment device and inum are sent as 4-byte integers, which
289 will probably cause problems on large filesystems. On Linux the
290 kernel uses 64-bit ino_t's internally, and people will soon have
291 filesystems big enough to use them. We ought to follow NFS4 in
292 using 64-bit device and inode identification, perhaps with a
293 protocol version bump.
295 Once we've seen all the names for a particular file, we no longer
296 need to think about it and we can deallocate the memory.
298 We can also have the case where there are links to a file that are
299 not in the tree being transferred. There's nothing we can do about
300 that. Because we rename the destination into place after writing,
301 any hardlinks to the old file are always going to be orphaned. In
302 fact that is almost necessary because otherwise we'd get really
303 confused if we were generating checksums for one name of a file and
306 At the moment the code seems to make a whole second copy of the file
307 list, which seems unnecessary.
309 We should have a test case that exercises hard links. Since it
310 might be hard to compare ./tls output where the inodes change we
311 might need a little program to check whether several names refer to
316 Perhaps put back the old socket code; if on a machine that does not
317 properly support the getaddrinfo API, then use it. This is probably
318 much simpler than reimplementing it.
320 Alternatively, have two different files implementing the same
321 interface, and choose either the new or the old API. This is
322 probably necessary for systems that e.g. have IPv6, but
323 gethostbyaddr() can't handle it. The Linux manpage claims this is
326 This might get us working again on RedHat 5 and similar systems.
327 Although the Kame patch seems like a good idea, in fact it is a much
328 broader interface than the relatively narrow "open by name", "accept
329 and log" interface that rsync uses internally, and it has the
330 disadvantage of clashing with half-arsed implementations of the API.
332 Implement suggestions from http://www.kame.net/newsletter/19980604/
333 and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
335 If a host has multiple addresses, then listen try to connect to all
336 in order until we get through. (getaddrinfo may return multiple
337 addresses.) This is kind of implemented already.
339 Possibly also when starting as a server we may need to listen on
340 multiple passive addresses. This might be a bit harder, because we
341 may need to select on all of them. Hm.
343 Define a syntax for IPv6 literal addresses. Since they include
344 colons, they tend to break most naming systems, including ours.
345 Based on the HTTP IPv6 syntax, I think we should use
347 rsync://[::1]/foo/bar
350 which should just take a small change to the parser code.
355 If we hang or get SIGINT, then explain where we were up to. Perhaps
356 have a static buffer that contains the current function name, or
357 some kind of description of what we were trying to do. This is a
358 little easier on people than needing to run strace/truss.
360 "The dungeon collapses! You are killed." Rather than "unexpected
361 eof" give a message that is more detailed if possible and also more
364 If we get an error writing to a socket, then we should perhaps
365 continue trying to read to see if an error message comes across
366 explaining why the socket is closed. I'm not sure if this would
367 work, but it would certainly make our messages more helpful.
369 What happens if a directory is missing -x attributes. Do we lose
370 our load? (Debian #28416) Probably fixed now, but a test case
376 Device major/minor numbers should be at least 32 bits each. See
377 http://lists.samba.org/pipermail/rsync/2001-November/005357.html
379 Transfer ACLs. Need to think of a standard representation.
380 Probably better not to even try to convert between NT and POSIX.
381 Possibly can share some code with Samba.
385 With the current common --include '*/' --exclude '*' pattern, people
386 can end up with many empty directories. We might avoid this by
387 lazily creating such directories.
392 Perhaps don't use our own zlib.
396 - will automatically be up to date with bugfixes in zlib
398 - can leave it out for small rsync on e.g. recovery disks
400 - can use a shared library
402 - avoids people breaking rsync by trying to do this themselves and
405 Should we ship zlib for systems that don't have it, or require
406 people to install it separately?
408 Apparently this will make us incompatible with versions of rsync
409 that use the patched version of rsync. Probably the simplest way to
410 do this is to just disable gzip (with a warning) when talking to old
416 Perhaps flush stdout after each filename, so that people trying to
417 monitor progress in a log file can do so more easily. See
418 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
420 At the connections that just get a list of modules are not logged,
423 If a child of the rsync daemon dies with a signal, we should notice
424 that when we reap it and log a message.
426 Keep stderr and stdout properly separated (Debian #23626)
428 After we get the @RSYNCD greeting from the server, we know it's
429 version but we have not yet sent the command line, so we could just
430 remove the -z option if the server is too old.
432 For ssh invocation it's not so simple, because we actually use the
433 command line to start the remote process. However, we only actually
434 do compression in token.c, and we could therefore once we discover
435 the remote version emit an error if it's too old. I'm not sure if
436 that's a good tradeoff or not.
441 There are already some patches to do this.
445 Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
446 HTTP Basic Proxy-Authentication.
448 Multiple schemes are possible, up to and including the insanity that
449 is NTLM, but Basic probably covers most cases.
453 Add --with-socks, and then perhaps a command-line option to put them
454 on or off. This might be more reliable than LD_PRELOAD hacks.
458 rsync to a FAT partition on a Unix machine doesn't work very well
459 at the moment. I think we get errors about invalid filenames and
460 perhaps also trying to do atomic renames.
462 I guess the code to do this is currently #ifdef'd on Windows; perhaps
463 we ought to intelligently fall back to it on Unix too.
468 <Rasmus> mbp: hey, how about an rsync option that just gives you the
469 summary without the list of files? And perhaps gives more
470 information like the number of new files, number of changed,
472 <mbp> Rasmus: nice idea
473 <mbp> there is --stats
474 <mbp> but at the moment it's very tridge-oriented
475 <mbp> rather than user-friendly
476 <mbp> it would be nice to improve it
477 <mbp> that would also work well with --dryrun
481 Rather than storing the file list in memory, store it in a TDB.
483 This *might* make memory usage lower while building the file list.
485 Hashtable lookup will mean files are not transmitted in order,
488 This would neatly eliminate one of the major post-fork shared data
494 On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote:
495 > If we would add an option to do that functionality, I would vote for one
496 > that was more general which could mask off any set of permission bits and
497 > possibly add any set of bits. Perhaps a chmod-like syntax if it could be
498 > implemented simply.
500 I think that would be good too. For example, people uploading files
501 to a web server might like to say
503 rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
505 Ideally the patch would implement as many of the gnu chmod semantics
506 as possible. I think the mode parser should be a separate function
507 that passes back something like (mask,set) description to the rest of
508 the program. For bonus points there would be a test case for the
511 Possibly also --chown
518 Allow people to specify the diff command. (Might want to use wdiff,
521 Just diff the temporary file with the destination file, and delete
522 the tmp file rather than moving it into place.
524 Interaction with --partial.
526 Security interactions with daemon mode?
528 (Suggestion from david.e.sewell)
531 Incorrect timestamps (Debian #100295)
533 A bit hard to believe, but apparently it happens.
536 Check "refuse options works"
538 We need a test case for this...
540 Was this broken when we changed to popt?
543 PERFORMANCE ----------------------------------------------------------
547 If we're doing a local transfer, or using -W, then perhaps don't
548 send the file checksum. If we're doing a local transfer, then
549 calculating MD4 checksums uses 90% of CPU and is unlikely to be
552 Indeed for transfers over zlib or ssh we can also rely on the
553 transport to have quite strong protection against corruption.
555 Perhaps we should have an option to disable this, analogous to
556 --whole-file, although it would default to disabled. The file
557 checksum takes up a definite space in the protocol -- we can either
558 set it to 0, or perhaps just leave it out.
562 Perhaps borrow an assembler MD4 from someone?
564 Make sure we call MD4 with properly-sized blocks whenever possible
565 to avoid copying into the residue region?
569 Test whether this is actually faster than just using malloc(). If
570 it's not (anymore), throw it out.
573 PLATFORMS ------------------------------------------------------------
577 Don't detach, because this messes up --srvany.
579 http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html
581 According to "Effective TCP/IP Programming" (??) close() on a socket
582 has incorrect behaviour on Windows -- it sends a RST packet to the
583 other side, which gives a "connection reset by peer" error. On that
584 platform we should probably do shutdown() instead. However, on Unix
585 we are correct to call close(), because shutdown() discards
589 DEVELOPMENT ----------------------------------------------------------
593 Build rsync with SPLINT to try to find security holes. Add
594 annotations as necessary. Keep track of the number of warnings
595 found initially, and see how many of them are real bugs, or real
596 security bugs. Knowing the percentage of likely hits would be
597 really interesting for other projects.
601 Something that just keeps running rsync continuously over a data set
602 likely to generate problems.
606 Run current rsync versions against significant past releases.
610 jra recommends Valgrind:
612 http://devel-home.kde.org/~sewardj/
618 Build tar file; upload
620 Send announcement to mailing list and c.o.l.a.
622 Make freshmeat announcement
628 TESTING --------------------------------------------------------------
632 Part of the regression suite should be making sure that we don't
633 break backwards compatibility: old clients vs new servers and so
634 on. Ideally we would test both up and down from the current release
637 We might need to omit broken old versions, or versions in which
638 particular functionality is broken
640 It might be sufficient to test downloads from well-known public
641 rsync servers running different versions of rsync. This will give
642 some testing and also be the most common case for having different
643 versions and not being able to upgrade.
646 Test on kernel source
648 Download all versions of kernel; unpack, sync between them. Also
649 sync between uncompressed tarballs. Compare directories after
652 Use local mode; ssh; daemon; --whole-file and --no-whole-file.
654 Use awk to pull out the 'speedup' number for each transfer. Make
660 Sparse and non-sparse
664 Insert bytes, delete bytes, swap blocks, ...
666 configure option to enable dangerous tests
668 If tests are skipped, say why.
670 Test daemon feature to disallow particular options.
672 Pipe program that makes slow/jerky connections.
674 Versions of read() and write() that corrupt the stream, or abruptly fail
676 Separate makefile target to run rough tests -- or perhaps just run
679 Test "refuse options" works
681 What about for --recursive?
683 If you specify an unrecognized option here, you should get an error.
686 DOCUMENTATION --------------------------------------------------------
690 Keep list of open issues and todos on the web site
692 Update web site from CVS
695 Perhaps redo manual as SGML
697 The man page is getting rather large, and there is more information
698 that ought to be added.
700 TexInfo source is probably a dying format.
702 Linuxdoc looks like the most likely contender. I know DocBook is
703 favoured by some people, but it's so bloody verbose, even with emacs
707 BUILD FARM -----------------------------------------------------------
711 AMDAHL UTS (Dave Dykstra)
713 Cygwin (on different versions of Win32?)
715 HP-UX variants (via HP?)
720 LOGGING --------------------------------------------------------------
722 Perhaps flush stdout after each filename, so that people trying to
723 monitor progress in a log file can do so more easily. See
724 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
726 At the connections that just get a list of modules are not logged,
729 If a child of the rsync daemon dies with a signal, we should notice
730 that when we reap it and log a message.
732 Keep stderr and stdout properly separated (Debian #23626)
734 Use a separate function for reporting errors; prefix it with
735 "rsync:" or "rsync(remote)", or perhaps even "rsync(local
740 Indicate whether files are new, updated, or deleted
742 At end of transfer, show how many files were or were not transferred
747 Explain *why* every file is transferred or not (e.g. "local mtime
748 123123 newer than 1283198")
753 Add an rsyncd.conf parameter to turn on debugging on the server.
757 NICE -----------------------------------------------------------------
759 --no-detach and --no-fork options
761 Very useful for debugging. Also good when running under a
762 daemon-monitoring process that tries to restart the service when the
765 hang/timeout friendliness
769 Change to using gettext(). Probably need to ship this for platforms
772 Solicit translations.
774 Does anyone care? Before we bother modifying the code, we ought to
775 get the manual translated first, because that's possibly more useful
776 and at any rate demonstrates desire.
780 Write a small emulation of interactive ftp as a Pythonn program
781 that calls rsync. Commands such as "cd", "ls", "ls *.c" etc map
782 fairly directly into rsync commands: it just needs to remember the
783 current host, directory and so on. We can probably even do
784 completion of remote filenames.
787 RELATED PROJECTS -----------------------------------------------------
789 http://rsync.samba.org/rsync-and-debian/
793 Exhaustive, tortuous testing
797 rsyncsplit as alternative to real integration with gzip?
799 reverse rsync over HTTP Range
801 Goswin Brederlow suggested this on Debian; I think tridge and I
802 talked about it previous in relation to rproxy.