3 BUGS ---------------------------------------------------------------
5 There seems to be a bug with hardlinks
7 mbp/2 build$ ls -l /tmp/a /tmp/b -i
10 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
11 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
12 2568307 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
13 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
14 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
15 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
16 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
17 2568310 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
21 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
22 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
23 2568309 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
24 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
25 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
26 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
27 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
28 2568311 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
29 mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
30 building file list ... done
31 created directory /tmp/b
37 wrote 350 bytes read 52 bytes 804.00 bytes/sec
38 total size is 232 speedup is 0.58
39 mbp/2 build$ rm -r /tmp/b
40 mbp/2 build$ ls -l /tmp/b
41 ls: /tmp/b: No such file or directory
42 mbp/2 build$ rm -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
43 rm: cannot remove `/tmp/b': No such file or directory
44 mbp/2 build$ rm -f -r /tmp/b && ./rsync -avH /tmp/a/ /tmp/b
45 building file list ... done
46 created directory /tmp/b
52 wrote 350 bytes read 52 bytes 804.00 bytes/sec
53 total size is 232 speedup is 0.58
54 mbp/2 build$ ls -l /tmp/b
56 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
57 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
58 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
59 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
60 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
61 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
62 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
63 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
64 mbp/2 build$ ls -l /tmp/a
66 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a1
67 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a2
68 -rw-rw-r-- 3 mbp mbp 29 Mar 25 17:30 a3
69 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a4
70 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 a5
71 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b1
72 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b2
73 -rw-rw-r-- 5 mbp mbp 29 Mar 25 17:30 b3
76 Progress indicator can produce corrupt output when transferring directories:
79 main/binary-arm/admin/
81 main/binary-arm/comm/8.56kB/s 0:00:52
82 main/binary-arm/devel/
84 main/binary-arm/editors/
85 main/binary-arm/electronics/s 0:00:53
86 main/binary-arm/games/
87 main/binary-arm/graphics/
88 main/binary-arm/hamradio/
89 main/binary-arm/interpreters/
90 main/binary-arm/libs/6.61kB/s 0:00:54
96 I don't think we handle this properly on systems that don't have the
100 Part of the regression suite should be making sure that we don't
101 break backwards compatibility: old clients vs new servers and so
102 on. Ideally we would test the cross product of versions.
104 It might be sufficient to test downloads from well-known public
105 rsync servers running different versions of rsync. This will give
106 some testing and also be the most common case for having different
107 versions and not being able to upgrade.
110 DAEMON --------------------------------------------------------------
112 server-imposed bandwidth limits
116 There are already some patches to do this.
118 BitKeeper uses a server whose login shell is set to bkd. That's
119 probably a reasonable approach.
122 FEATURES ------------------------------------------------------------
125 --dry-run is insufficiently dry
127 Mark Santcroos points out that -n fails to list files which have
128 only metadata changes, though it probably should.
130 There may be a Debian bug about this as well.
135 If the platform doesn't support it, then don't even try.
137 If running as non-root, then don't fail, just give a warning.
138 (There was a thread about this a while ago?)
140 http://lists.samba.org/pipermail/rsync/2001-August/thread.html
141 http://lists.samba.org/pipermail/rsync/2001-September/thread.html
146 Avoids traversal. Better option than a pile of --include statements
147 for people who want to generate the file list using a find(1)
153 Perhaps allow supplementary groups to be specified in rsyncd.conf;
154 then make the first one the primary gid and all the rest be
158 File list structure in memory
160 Rather than one big array, perhaps have a tree in memory mirroring
163 This might make sorting much faster! (I'm not sure it's a big CPU
166 It might also reduce memory use in storing repeated directory names
167 -- again I'm not sure this is a problem.
171 Traverse just one directory at a time. Tridge says it's possible.
173 At the moment rsync reads the whole file list into memory at the
174 start, which makes us use a lot of memory and also not pipeline
175 network access as much as we could.
178 Handling duplicate names
180 We need to be careful of duplicate names getting into the file list.
181 See clean_flist(). This could happen if multiple arguments include
184 I think duplicates are only a problem if they're both flowing
185 through the pipeline at the same time. For example we might have
186 updated the first occurrence after reading the checksums for the
187 second. So possibly we just need to make sure that we don't have
188 both in the pipeline at the same time.
190 Possibly if we did one directory at a time that would be sufficient.
192 Alternatively we could pre-process the arguments to make sure no
193 duplicates will ever be inserted. There could be some bad cases
194 when we're collapsing symlinks.
196 We could have a hash table.
198 The root of the problem is that we do not want more than one file
199 list entry referring to the same file. At first glance there are
200 several ways this could happen: symlinks, hardlinks, and repeated
201 names on the command line.
203 If names are repeated on the command line, they may be present in
204 different forms, perhaps by traversing directory paths in different
205 ways, traversing paths including symlinks. Also we need to allow
206 for expansion of globs by rsync.
208 At the moment, clean_flist() requires having the entire file list in
209 memory. Duplicate names are detected just by a string comparison.
211 We don't need to worry about hard links causing duplicates because
212 files are never updated in place. Similarly for symlinks.
214 I think even if we're using a different symlink mode we don't need
217 Unless we're really clever this will introduce a protocol
218 incompatibility, so we need to be able to accept the old format as
224 At exit, show how much memory was used for the file list, etc.
226 Also we do a wierd exponential-growth allocation in flist.c. I'm
227 not sure this makes sense with modern mallocs. At any rate it will
228 make us allocate a huge amount of memory for large file lists.
233 At the moment hardlink handling is very expensive, so it's off by
234 default. It does not need to be so.
236 Since most of the solutions are rather intertwined with the file
237 list it is probably better to fix that first, although fixing
238 hardlinks is possibly simpler.
240 We can rule out hardlinked directories since they will probably
241 screw us up in all kinds of ways. They simply should not be used.
243 At the moment rsync only cares about hardlinks to regular files. I
244 guess you could also use them for sockets, devices and other beasts,
245 but I have not seen them.
247 When trying to reproduce hard links, we only need to worry about
248 files that have more than one name (nlinks>1 && !S_ISDIR).
250 The basic point of this is to discover alternate names that refer to
251 the same file. All operations, including creating the file and
252 writing modifications to it need only to be done for the first name.
253 For all later names, we just create the link and then leave it
256 If hard links are to be preserved:
258 Before the generator/receiver fork, the list of files is received
259 from the sender (recv_file_list), and a table for detecting hard
262 The generator looks for hard links within the file list and does
263 not send checksums for them, though it does send other metadata.
265 The sender sends the device number and inode with file entries, so
266 that files are uniquely identified.
268 The receiver goes through and creates hard links (do_hard_links)
269 after all data has been written, but before directory permissions
272 At the moment device and inum are sent as 4-byte integers, which
273 will probably cause problems on large filesystems. On Linux the
274 kernel uses 64-bit ino_t's internally, and people will soon have
275 filesystems big enough to use them. We ought to follow NFS4 in
276 using 64-bit device and inode identification, perhaps with a
277 protocol version bump.
279 Once we've seen all the names for a particular file, we no longer
280 need to think about it and we can deallocate the memory.
282 We can also have the case where there are links to a file that are
283 not in the tree being transferred. There's nothing we can do about
284 that. Because we rename the destination into place after writing,
285 any hardlinks to the old file are always going to be orphaned. In
286 fact that is almost necessary because otherwise we'd get really
287 confused if we were generating checksums for one name of a file and
290 At the moment the code seems to make a whole second copy of the file
291 list, which seems unnecessary.
293 We should have a test case that exercises hard links. Since it
294 might be hard to compare ./tls output where the inodes change we
295 might need a little program to check whether several names refer to
300 Implement suggestions from http://www.kame.net/newsletter/19980604/
301 and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
303 If a host has multiple addresses, then listen try to connect to all
304 in order until we get through. (getaddrinfo may return multiple
305 addresses.) This is kind of implemented already.
307 Possibly also when starting as a server we may need to listen on
308 multiple passive addresses. This might be a bit harder, because we
309 may need to select on all of them. Hm.
311 Define a syntax for IPv6 literal addresses. Since they include
312 colons, they tend to break most naming systems, including ours.
313 Based on the HTTP IPv6 syntax, I think we should use
315 rsync://[::1]/foo/bar
318 which should just take a small change to the parser code.
323 If we hang or get SIGINT, then explain where we were up to. Perhaps
324 have a static buffer that contains the current function name, or
325 some kind of description of what we were trying to do. This is a
326 little easier on people than needing to run strace/truss.
328 "The dungeon collapses! You are killed." Rather than "unexpected
329 eof" give a message that is more detailed if possible and also more
332 If we get an error writing to a socket, then we should perhaps
333 continue trying to read to see if an error message comes across
334 explaining why the socket is closed. I'm not sure if this would
335 work, but it would certainly make our messages more helpful.
337 What happens if a directory is missing -x attributes. Do we lose
338 our load? (Debian #28416) Probably fixed now, but a test case
344 Device major/minor numbers should be at least 32 bits each. See
345 http://lists.samba.org/pipermail/rsync/2001-November/005357.html
347 Transfer ACLs. Need to think of a standard representation.
348 Probably better not to even try to convert between NT and POSIX.
349 Possibly can share some code with Samba.
353 With the current common --include '*/' --exclude '*' pattern, people
354 can end up with many empty directories. We might avoid this by
355 lazily creating such directories.
360 Perhaps don't use our own zlib.
364 - will automatically be up to date with bugfixes in zlib
366 - can leave it out for small rsync on e.g. recovery disks
368 - can use a shared library
370 - avoids people breaking rsync by trying to do this themselves and
373 Should we ship zlib for systems that don't have it, or require
374 people to install it separately?
376 Apparently this will make us incompatible with versions of rsync
377 that use the patched version of rsync. Probably the simplest way to
378 do this is to just disable gzip (with a warning) when talking to old
384 Perhaps flush stdout after each filename, so that people trying to
385 monitor progress in a log file can do so more easily. See
386 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
388 At the connections that just get a list of modules are not logged,
391 If a child of the rsync daemon dies with a signal, we should notice
392 that when we reap it and log a message.
394 Keep stderr and stdout properly separated (Debian #23626)
396 After we get the @RSYNCD greeting from the server, we know it's
397 version but we have not yet sent the command line, so we could just
398 remove the -z option if the server is too old.
400 For ssh invocation it's not so simple, because we actually use the
401 command line to start the remote process. However, we only actually
402 do compression in token.c, and we could therefore once we discover
403 the remote version emit an error if it's too old. I'm not sure if
404 that's a good tradeoff or not.
409 There are already some patches to do this.
413 Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
414 HTTP Basic Proxy-Authentication.
416 Multiple schemes are possible, up to and including the insanity that
417 is NTLM, but Basic probably covers most cases.
421 Add --with-socks, and then perhaps a command-line option to put them
422 on or off. This might be more reliable than LD_PRELOAD hacks.
426 rsync to a FAT partition on a Unix machine doesn't work very well
427 at the moment. I think we get errors about invalid filenames and
428 perhaps also trying to do atomic renames.
430 I guess the code to do this is currently #ifdef'd on Windows; perhaps
431 we ought to intelligently fall back to it on Unix too.
436 <Rasmus> mbp: hey, how about an rsync option that just gives you the
437 summary without the list of files? And perhaps gives more
438 information like the number of new files, number of changed,
440 <mbp> Rasmus: nice idea
441 <mbp> there is --stats
442 <mbp> but at the moment it's very tridge-oriented
443 <mbp> rather than user-friendly
444 <mbp> it would be nice to improve it
445 <mbp> that would also work well with --dryrun
449 Rather than storing the file list in memory, store it in a TDB.
451 This *might* make memory usage lower while building the file list.
453 Hashtable lookup will mean files are not transmitted in order,
456 This would neatly eliminate one of the major post-fork shared data
462 On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote:
463 > If we would add an option to do that functionality, I would vote for one
464 > that was more general which could mask off any set of permission bits and
465 > possibly add any set of bits. Perhaps a chmod-like syntax if it could be
466 > implemented simply.
468 I think that would be good too. For example, people uploading files
469 to a web server might like to say
471 rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
473 Ideally the patch would implement as many of the gnu chmod semantics
474 as possible. I think the mode parser should be a separate function
475 that passes back something like (mask,set) description to the rest of
476 the program. For bonus points there would be a test case for the
484 Allow people to specify the diff command. (Might want to use wdiff,
487 Just diff the temporary file with the destination file, and delete
488 the tmp file rather than moving it into place.
490 Interaction with --partial.
492 Security interactions with daemon mode?
494 (Suggestion from david.e.sewell)
497 Incorrect timestamps (Debian #100295)
499 A bit hard to believe, but apparently it happens.
502 Check "refuse options works"
504 We need a test case for this...
506 Was this broken when we changed to popt?
509 PERFORMANCE ----------------------------------------------------------
513 If we're doing a local transfer, or using -W, then perhaps don't
514 send the file checksum. If we're doing a local transfer, then
515 calculating MD4 checksums uses 90% of CPU and is unlikely to be
518 Indeed for transfers over zlib or ssh we can also rely on the
519 transport to have quite strong protection against corruption.
521 Perhaps we should have an option to disable this, analogous to
522 --whole-file, although it would default to disabled. The file
523 checksum takes up a definite space in the protocol -- we can either
524 set it to 0, or perhaps just leave it out.
528 Perhaps borrow an assembler MD4 from someone?
530 Make sure we call MD4 with properly-sized blocks whenever possible
531 to avoid copying into the residue region?
535 Test whether this is actually faster than just using malloc(). If
536 it's not (anymore), throw it out.
539 PLATFORMS ------------------------------------------------------------
543 Don't detach, because this messes up --srvany.
545 http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html
547 According to "Effective TCP/IP Programming" (??) close() on a socket
548 has incorrect behaviour on Windows -- it sends a RST packet to the
549 other side, which gives a "connection reset by peer" error. On that
550 platform we should probably do shutdown() instead. However, on Unix
551 we are correct to call close(), because shutdown() discards
555 DEVELOPMENT ----------------------------------------------------------
559 Build rsync with SPLINT to try to find security holes. Add
560 annotations as necessary. Keep track of the number of warnings
561 found initially, and see how many of them are real bugs, or real
562 security bugs. Knowing the percentage of likely hits would be
563 really interesting for other projects.
567 Something that just keeps running rsync continuously over a data set
568 likely to generate problems.
572 Run current rsync versions against significant past releases.
576 jra recommends Valgrind:
578 http://devel-home.kde.org/~sewardj/
584 Build tar file; upload
586 Send announcement to mailing list and c.o.l.a.
588 Make freshmeat announcement
594 TESTING --------------------------------------------------------------
598 Part of the regression suite should be making sure that we don't
599 break backwards compatibility: old clients vs new servers and so
600 on. Ideally we would test both up and down from the current release
603 We might need to omit broken old versions, or versions in which
604 particular functionality is broken
606 It might be sufficient to test downloads from well-known public
607 rsync servers running different versions of rsync. This will give
608 some testing and also be the most common case for having different
609 versions and not being able to upgrade.
612 Test on kernel source
614 Download all versions of kernel; unpack, sync between them. Also
615 sync between uncompressed tarballs. Compare directories after
618 Use local mode; ssh; daemon; --whole-file and --no-whole-file.
620 Use awk to pull out the 'speedup' number for each transfer. Make
626 Sparse and non-sparse
630 Insert bytes, delete bytes, swap blocks, ...
632 configure option to enable dangerous tests
634 If tests are skipped, say why.
636 Test daemon feature to disallow particular options.
638 Pipe program that makes slow/jerky connections.
640 Versions of read() and write() that corrupt the stream, or abruptly fail
642 Separate makefile target to run rough tests -- or perhaps just run
645 Test "refuse options" works
647 What about for --recursive?
649 If you specify an unrecognized option here, you should get an error.
652 DOCUMENTATION --------------------------------------------------------
656 Keep list of open issues and todos on the web site
658 Update web site from CVS
661 Perhaps redo manual as SGML
663 The man page is getting rather large, and there is more information
664 that ought to be added.
666 TexInfo source is probably a dying format.
668 Linuxdoc looks like the most likely contender. I know DocBook is
669 favoured by some people, but it's so bloody verbose, even with emacs
673 BUILD FARM -----------------------------------------------------------
677 AMDAHL UTS (Dave Dykstra)
679 Cygwin (on different versions of Win32?)
681 HP-UX variants (via HP?)
686 LOGGING --------------------------------------------------------------
688 Perhaps flush stdout after each filename, so that people trying to
689 monitor progress in a log file can do so more easily. See
690 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
692 At the connections that just get a list of modules are not logged,
695 If a child of the rsync daemon dies with a signal, we should notice
696 that when we reap it and log a message.
698 Keep stderr and stdout properly separated (Debian #23626)
700 Use a separate function for reporting errors; prefix it with
701 "rsync:" or "rsync(remote)", or perhaps even "rsync(local
706 Indicate whether files are new, updated, or deleted
708 At end of transfer, show how many files were or were not transferred
713 Explain *why* every file is transferred or not (e.g. "local mtime
714 123123 newer than 1283198")
719 Add an rsyncd.conf parameter to turn on debugging on the server.
723 NICE -----------------------------------------------------------------
725 --no-detach and --no-fork options
727 Very useful for debugging. Also good when running under a
728 daemon-monitoring process that tries to restart the service when the
731 hang/timeout friendliness
735 Change to using gettext(). Probably need to ship this for platforms
738 Solicit translations.
740 Does anyone care? Before we bother modifying the code, we ought to
741 get the manual translated first, because that's possibly more useful
742 and at any rate demonstrates desire.
746 Write a small emulation of interactive ftp as a Pythonn program
747 that calls rsync. Commands such as "cd", "ls", "ls *.c" etc map
748 fairly directly into rsync commands: it just needs to remember the
749 current host, directory and so on. We can probably even do
750 completion of remote filenames.
753 RELATED PROJECTS -----------------------------------------------------
755 http://rsync.samba.org/rsync-and-debian/
759 Exhaustive, tortuous testing
763 rsyncsplit as alternative to real integration with gzip?
765 reverse rsync over HTTP Range
767 Goswin Brederlow suggested this on Debian; I think tridge and I
768 talked about it previous in relation to rproxy.