- A list of all files are held in memory for the entire transfer,
which cripples scalability to large file trees
- A list of all files are held in memory for the entire transfer,
which cripples scalability to large file trees
- The remote rsync can be wrapped by some other program, such as in
tridge's rsync-mail scripts. The general feature of sending and
- The remote rsync can be wrapped by some other program, such as in
tridge's rsync-mail scripts. The general feature of sending and
Alternatively, as long as transfers are idempotent, we can just
restart the whole thing. [NFSv4]
Alternatively, as long as transfers are idempotent, we can just
restart the whole thing. [NFSv4]
- Propagate atimes and do not modify them. This is very ugly on
Unix. It might be better to try to add O_NOATIME to kernels, and
- Propagate atimes and do not modify them. This is very ugly on
Unix. It might be better to try to add O_NOATIME to kernels, and
These might have a severe impact on the protocol, and are not
clearly in our core requirements. It looks like in many of them
These might have a severe impact on the protocol, and are not
clearly in our core requirements. It looks like in many of them
- Transport over UDP multicast. The hard part is handling multiple
destinations which have different basis files. We can look at
- Transport over UDP multicast. The hard part is handling multiple
destinations which have different basis files. We can look at
- If we start from scratch, it can be documented as we go, and we
can avoid design decisions that make the protocol complex or
- If we start from scratch, it can be documented as we go, and we
can avoid design decisions that make the protocol complex or
- We can do nonblocking network IO, but not so for disk.
- It makes sense to on the destination be generating signatures and
- We can do nonblocking network IO, but not so for disk.
- It makes sense to on the destination be generating signatures and
- Can structure this with nonblocking, threads, separate processes,
etc.
- Can structure this with nonblocking, threads, separate processes,
etc.
If migrating a whole tree, then we could do a nearly-atomic rename
at the end. This ties in to having separate basis and destination
If migrating a whole tree, then we could do a nearly-atomic rename
at the end. This ties in to having separate basis and destination
There's no way in Unix to replace a whole set of files atomically.
However, if we get them all onto the destination machine and then do
There's no way in Unix to replace a whole set of files atomically.
However, if we get them all onto the destination machine and then do
We should aim to work well on machines in use in a year or two.
That probably means transfers of many millions of files in one
batch, and gigabytes or terabytes of data.
We should aim to work well on machines in use in a year or two.
That probably means transfers of many millions of files in one
batch, and gigabytes or terabytes of data.
- http://freshmeat.net/search/?site=Freshmeat&q=mirror§ion=projects
- BitTorrent -- p2p mirroring
- http://freshmeat.net/search/?site=Freshmeat&q=mirror§ion=projects
- BitTorrent -- p2p mirroring