3 URGENT ---------------------------------------------------------------
6 IMPORTANT ------------------------------------------------------------
11 If the platform doesn't support it, then don't even try.
13 If running as non-root, then don't fail, just give a warning.
14 (There was a thread about this a while ago?)
16 http://lists.samba.org/pipermail/rsync/2001-August/thread.html
17 http://lists.samba.org/pipermail/rsync/2001-September/thread.html
21 Avoids traversal. Better option than a pile of --include statements
22 for people who want to generate the file list using a find(1)
25 File list structure in memory
27 Rather than one big array, perhaps have a tree in memory mirroring
30 This might make sorting much faster! (I'm not sure it's a big CPU
33 It might also reduce memory use in storing repeated directory names
34 -- again I'm not sure this is a problem.
38 Traverse just one directory at a time. Tridge says it's possible.
40 At the moment rsync reads the whole file list into memory at the
41 start, which makes us use a lot of memory and also not pipeline
42 network access as much as we could.
45 Handling duplicate names
47 We need to be careful of duplicate names getting into the file list.
48 See clean_flist(). This could happen if multiple arguments include
51 I think duplicates are only a problem if they're both flowing
52 through the pipeline at the same time. For example we might have
53 updated the first occurrence after reading the checksums for the
54 second. So possibly we just need to make sure that we don't have
55 both in the pipeline at the same time.
57 Possibly if we did one directory at a time that would be sufficient.
59 Alternatively we could pre-process the arguments to make sure no
60 duplicates will ever be inserted. There could be some bad cases
61 when we're collapsing symlinks.
63 We could have a hash table.
65 The root of the problem is that we do not want more than one file
66 list entry referring to the same file. At first glance there are
67 several ways this could happen: symlinks, hardlinks, and repeated
68 names on the command line.
70 If names are repeated on the command line, they may be present in
71 different forms, perhaps by traversing directory paths in different
72 ways, traversing paths including symlinks. Also we need to allow
73 for expansion of globs by rsync.
75 At the moment, clean_flist() requires having the entire file list in
76 memory. Duplicate names are detected just by a string comparison.
78 We don't need to worry about hard links causing duplicates because
79 files are never updated in place. Similarly for symlinks.
81 I think even if we're using a different symlink mode we don't need
84 Unless we're really clever this will introduce a protocol
85 incompatibility, so we need to be able to accept the old format as
91 At exit, show how much memory was used for the file list, etc.
93 Also we do a wierd exponential-growth allocation in flist.c. I'm
94 not sure this makes sense with modern mallocs. At any rate it will
95 make us allocate a huge amount of memory for large file lists.
100 At the moment hardlink handling is very expensive, so it's off by
101 default. It does not need to be so.
103 Since most of the solutions are rather intertwined with the file
104 list it is probably better to fix that first, although fixing
105 hardlinks is possibly simpler.
107 We can rule out hardlinked directories since they will probably
108 screw us up in all kinds of ways. They simply should not be used.
110 At the moment rsync only cares about hardlinks to regular files. I
111 guess you could also use them for sockets, devices and other beasts,
112 but I have not seen them.
114 When trying to reproduce hard links, we only need to worry about
115 files that have more than one name (nlinks>1 && !S_ISDIR).
117 The basic point of this is to discover alternate names that refer to
118 the same file. All operations, including creating the file and
119 writing modifications to it need only to be done for the first name.
120 For all later names, we just create the link and then leave it
123 If hard links are to be preserved:
125 Before the generator/receiver fork, the list of files is received
126 from the sender (recv_file_list), and a table for detecting hard
129 The generator looks for hard links within the file list and does
130 not send checksums for them, though it does send other metadata.
132 The sender sends the device number and inode with file entries, so
133 that files are uniquely identified.
135 The receiver goes through and creates hard links (do_hard_links)
136 after all data has been written, but before directory permissions
139 At the moment device and inum are sent as 4-byte integers, which
140 will probably cause problems on large filesystems. On Linux the
141 kernel uses 64-bit ino_t's internally, and people will soon have
142 filesystems big enough to use them. We ought to follow NFS4 in
143 using 64-bit device and inode identification, perhaps with a
144 protocol version bump.
146 Once we've seen all the names for a particular file, we no longer
147 need to think about it and we can deallocate the memory.
149 We can also have the case where there are links to a file that are
150 not in the tree being transferred. There's nothing we can do about
151 that. Because we rename the destination into place after writing,
152 any hardlinks to the old file are always going to be orphaned. In
153 fact that is almost necessary because otherwise we'd get really
154 confused if we were generating checksums for one name of a file and
157 At the moment the code seems to make a whole second copy of the file
158 list, which seems unnecessary.
160 We should have a test case that exercises hard links. Since it
161 might be hard to compare ./tls output where the inodes change we
162 might need a little program to check whether several names refer to
167 Implement suggestions from http://www.kame.net/newsletter/19980604/
168 and ftp://ftp.iij.ad.jp/pub/RFC/rfc2553.txt
170 If a host has multiple addresses, then listen try to connect to all
171 in order until we get through. (getaddrinfo may return multiple
172 addresses.) This is kind of implemented already.
174 Possibly also when starting as a server we may need to listen on
175 multiple passive addresses. This might be a bit harder, because we
176 may need to select on all of them. Hm.
178 Define a syntax for IPv6 literal addresses. Since they include
179 colons, they tend to break most naming systems, including ours.
180 Based on the HTTP IPv6 syntax, I think we should use
182 rsync://[::1]/foo/bar
185 which should just take a small change to the parser code.
190 If we hang or get SIGINT, then explain where we were up to. Perhaps
191 have a static buffer that contains the current function name, or
192 some kind of description of what we were trying to do. This is a
193 little easier on people than needing to run strace/truss.
195 "The dungeon collapses! You are killed." Rather than "unexpected
196 eof" give a message that is more detailed if possible and also more
199 If we get an error writing to a socket, then we should perhaps
200 continue trying to read to see if an error message comes across
201 explaining why the socket is closed. I'm not sure if this would
202 work, but it would certainly make our messages more helpful.
204 What happens if a directory is missing -x attributes. Do we lose
205 our load? (Debian #28416) Probably fixed now, but a test case
211 Device major/minor numbers should be at least 32 bits each. See
212 http://lists.samba.org/pipermail/rsync/2001-November/005357.html
214 Transfer ACLs. Need to think of a standard representation.
215 Probably better not to even try to convert between NT and POSIX.
216 Possibly can share some code with Samba.
220 With the current common --include '*/' --exclude '*' pattern, people
221 can end up with many empty directories. We might avoid this by
222 lazily creating such directories.
227 Perhaps don't use our own zlib.
231 - will automatically be up to date with bugfixes in zlib
233 - can leave it out for small rsync on e.g. recovery disks
235 - can use a shared library
237 - avoids people breaking rsync by trying to do this themselves and
240 Should we ship zlib for systems that don't have it, or require
241 people to install it separately?
243 Apparently this will make us incompatible with versions of rsync
244 that use the patched version of rsync. Probably the simplest way to
245 do this is to just disable gzip (with a warning) when talking to old
251 Perhaps flush stdout after each filename, so that people trying to
252 monitor progress in a log file can do so more easily. See
253 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=48108
255 At the connections that just get a list of modules are not logged,
258 If a child of the rsync daemon dies with a signal, we should notice
259 that when we reap it and log a message.
261 Keep stderr and stdout properly separated (Debian #23626)
263 Use a separate function for reporting errors; prefix it with
264 "rsync:" or "rsync(remote)", or perhaps even "rsync(local
270 There are already some patches to do this.
274 Allow RSYNC_PROXY to be http://user:pass@proxy.foo:3128/, and do
275 HTTP Basic Proxy-Authentication.
277 Multiple schemes are possible, up to and including the insanity that
278 is NTLM, but Basic probably covers most cases.
282 Add --with-socks, and then perhaps a command-line option to put them
283 on or off. This might be more reliable than LD_PRELOAD hacks.
287 <Rasmus> mbp: hey, how about an rsync option that just gives you the
288 summary without the list of files? And perhaps gives more
289 information like the number of new files, number of changed,
291 <mbp> Rasmus: nice idea
292 <mbp> there is --stats
293 <mbp> but at the moment it's very tridge-oriented
294 <mbp> rather than user-friendly
295 <mbp> it would be nice to improve it
296 <mbp> that would also work well with --dryrun
300 Rather than storing the file list in memory, store it in a TDB.
302 This *might* make memory usage lower while building the file list.
304 Hashtable lookup will mean files are not transmitted in order,
307 This would neatly eliminate one of the major post-fork shared data
313 On 12 Mar 2002, Dave Dykstra <dwd@bell-labs.com> wrote:
314 > If we would add an option to do that functionality, I would vote for one
315 > that was more general which could mask off any set of permission bits and
316 > possibly add any set of bits. Perhaps a chmod-like syntax if it could be
317 > implemented simply.
319 I think that would be good too. For example, people uploading files
320 to a web server might like to say
322 rsync -avzP --chmod a+rX ./ sourcefrog.net:/home/www/sourcefrog/
324 Ideally the patch would implement as many of the gnu chmod semantics
325 as possible. I think the mode parser should be a separate function
326 that passes back something like (mask,set) description to the rest of
327 the program. For bonus points there would be a test case for the
335 Allow people to specify the diff command. (Might want to use wdiff,
338 Just diff the temporary file with the destination file, and delete
339 the tmp file rather than moving it into place.
341 Interaction with --partial.
343 Security interactions with daemon mode?
345 (Suggestion from david.e.sewell)
348 Incorrect timestamps (Debian #100295)
350 A bit hard to believe, but apparently it happens.
353 Check "refuse options works"
355 We need a test case for this...
357 Was this broken when we changed to popt?
362 Test whether this is actually faster than just using malloc(). If
363 it's not (anymore), throw it out.
367 PLATFORMS ------------------------------------------------------------
371 Don't detach, because this messes up --srvany.
373 http://sources.redhat.com/ml/cygwin/2001-08/msg00234.html
375 According to "Effective TCP/IP Programming" (??) close() on a socket
376 has incorrect behaviour on Windows -- it sends a RST packet to the
377 other side, which gives a "connection reset by peer" error. On that
378 platform we should probably do shutdown() instead. However, on Unix
379 we are correct to call close(), because shutdown() discards
382 DEVELOPMENT ----------------------------------------------------------
386 Build rsync with SPLINT to try to find security holes. Add
387 annotations as necessary. Keep track of the number of warnings
388 found initially, and see how many of them are real bugs, or real
389 security bugs. Knowing the percentage of likely hits would be
390 really interesting for other projects.
394 Something that just keeps running rsync continuously over a data set
395 likely to generate problems.
399 Run current rsync versions against significant past releases.
403 jra recommends Valgrind:
405 http://devel-home.kde.org/~sewardj/
407 TESTING --------------------------------------------------------------
411 Part of the regression suite should be making sure that we don't
412 break backwards compatibility: old clients vs new servers and so
413 on. Ideally we would test the cross product of versions.
415 It might be sufficient to test downloads from well-known public
416 rsync servers running different versions of rsync. This will give
417 some testing and also be the most common case for having different
418 versions and not being able to upgrade.
422 Sparse and non-sparse
426 Insert bytes, delete bytes, swap blocks, ...
428 configure option to enable dangerous tests
430 If tests are skipped, say why.
432 Test daemon feature to disallow particular options.
434 Pipe program that makes slow/jerky connections.
436 Versions of read() and write() that corrupt the stream, or abruptly fail
438 Separate makefile target to run rough tests -- or perhaps just run
442 DOCUMENTATION --------------------------------------------------------
446 Keep list of open issues and todos on the web site
448 Update web site from CVS
450 BUILD FARM -----------------------------------------------------------
454 AMDAHL UTS (Dave Dykstra)
456 Cygwin (on different versions of Win32?)
458 HP-UX variants (via HP?)
462 NICE -----------------------------------------------------------------
464 --no-detach and --no-fork options
466 Very useful for debugging. Also good when running under a
467 daemon-monitoring process that tries to restart the service when the
470 hang/timeout friendliness
474 Indicate whether files are new, updated, or deleted
476 At end of transfer, show how many files were or were not transferred
481 Change to using gettext(). Probably need to ship this for platforms
484 Solicit translations.
490 Write a small emulation of interactive ftp as a Pythonn program
491 that calls rsync. Commands such as "cd", "ls", "ls *.c" etc map
492 fairly directly into rsync commands: it just needs to remember the
493 current host, directory and so on. We can probably even do
494 completion of remote filenames.