After applying this patch and running configure, you MUST run this command before "make": make proto This patch adds the --filter option, which implements an improved set of include/exclude rules: . SINGLE-INSTANCE_MERGE_FILE : PER-DIRECTORY_MERGE_FILE - exclude-pattern + include-pattern Note that the prefix for a filter rule is NOT optional, and that the separating space can be replaced by an equal-sign (=) or an underscore (_), if desired. There are also optional modifiers that can be specified for the merge-file rules. A per-directory merge file is one that will be looked for in every sub-directory that rsync visits, and the rules found in that sub- directory's file will affect that dir and (if desired) its subdirs. For example: rsync -av --filter :_.rules from/ to The above will look for a file named ".rules" in every directory of the hierarchy that rsync visits, and it will filter names based on the rules found therein. If one of the .rules files contains this: + *.c : .rules2 . .rules3 - *.o - /foobar Then the file ".rules2" will also be read in from the current dir and all its subdirs. The file ".rules3" would just be read in from the current dir only. The exclusion of "foobar" will only happen in that .rules file's directory because the rule is anchored, which is one way to make a rule local instead of inherited (see also the 'n' modifier). ..wayne.. --- orig/clientserver.c 2005-01-01 21:11:00 +++ clientserver.c 2005-01-16 23:33:02 @@ -49,12 +49,14 @@ extern int no_detach; extern int default_af_hint; extern char *bind_address; extern struct exclude_list_struct server_exclude_list; -extern char *exclude_path_prefix; extern char *config_file; extern char *files_from; char *auth_user; +/* Length of lp_path() string when in daemon mode & not chrooted, else 0. */ +unsigned int module_dirlen = 0; + /** * Run a client connected to an rsyncd. The alternative to this * function for remote-shell connections is do_cmd(). @@ -304,26 +306,33 @@ static int rsync_module(int f_in, int f_ /* TODO: Perhaps take a list of gids, and make them into the * supplementary groups. */ - exclude_path_prefix = use_chroot? "" : lp_path(i); - if (*exclude_path_prefix == '/' && !exclude_path_prefix[1]) - exclude_path_prefix = ""; + if (use_chroot) { + module_dirlen = 0; + set_excludes_dir("/", 1); + } else { + module_dirlen = strlen(lp_path(i)); + set_excludes_dir(lp_path(i), module_dirlen); + } + + p = lp_filter(i); + add_exclude(&server_exclude_list, p, + XFLG_WORD_SPLIT | XFLG_ABS_PATH); p = lp_include_from(i); add_exclude_file(&server_exclude_list, p, - XFLG_FATAL_ERRORS | XFLG_DEF_INCLUDE); + XFLG_FATAL_ERRORS | XFLG_ABS_PATH | XFLG_DEF_INCLUDE); p = lp_include(i); add_exclude(&server_exclude_list, p, - XFLG_WORD_SPLIT | XFLG_DEF_INCLUDE); + XFLG_WORD_SPLIT | XFLG_ABS_PATH | XFLG_DEF_INCLUDE); p = lp_exclude_from(i); add_exclude_file(&server_exclude_list, p, - XFLG_FATAL_ERRORS); + XFLG_FATAL_ERRORS | XFLG_ABS_PATH | XFLG_DEF_EXCLUDE); p = lp_exclude(i); - add_exclude(&server_exclude_list, p, XFLG_WORD_SPLIT); - - exclude_path_prefix = NULL; + add_exclude(&server_exclude_list, p, + XFLG_WORD_SPLIT | XFLG_ABS_PATH | XFLG_DEF_EXCLUDE); log_init(); --- orig/exclude.c 2005-01-13 23:15:56 +++ exclude.c 2005-01-17 05:55:59 @@ -30,15 +30,68 @@ extern int verbose; extern int eol_nulls; extern int list_only; extern int recurse; +extern int io_error; +extern int sanitize_paths; +extern int protocol_version; extern char curr_dir[]; +extern unsigned int curr_dir_len; +extern unsigned int module_dirlen; struct exclude_list_struct exclude_list = { 0, 0, "" }; -struct exclude_list_struct local_exclude_list = { 0, 0, "per-dir .cvsignore " }; struct exclude_list_struct server_exclude_list = { 0, 0, "server " }; -char *exclude_path_prefix = NULL; -/** Build an exclude structure given an exclude pattern. */ +/* The dirbuf is set by push_local_excludes() to the current subdirectory + * relative to curr_dir that is being processed. The path always has a + * trailing slash appended, and the variable dirbuf_len contains the length + * of this path prefix. The path is always absolute. */ +static char dirbuf[MAXPATHLEN+1]; +static unsigned int dirbuf_len = 0; +static int dirbuf_depth; + +/* This is True when we're scanning parent dirs for per-dir merge-files. */ +static BOOL parent_dirscan = False; + +/* This array contains a list of all the currently active per-dir merge + * files. This makes it easier to save the appropriate values when we + * "push" down into each subdirectory. */ +static struct exclude_struct **mergelist_parents; +static int mergelist_cnt = 0; +static int mergelist_size = 0; + +/* Each exclude_list_struct describes a singly-linked list by keeping track + * of both the head and tail pointers. The list is slightly unusual in that + * a parent-dir's content can be appended to the end of the local list in a + * special way: the last item in the local list has its "next" pointer set + * to point to the inherited list, but the local list's tail pointer points + * at the end of the local list. Thus, if the local list is empty, the head + * will be pointing at the inherited content but the tail will be NULL. To + * help you visualize this, here are the possible list arrangements: + * + * Completely Empty Local Content Only + * ================================== ==================================== + * head -> NULL head -> Local1 -> Local2 -> NULL + * tail -> NULL tail -------------^ + * + * Inherited Content Only Both Local and Inherited Content + * ================================== ==================================== + * head -> Parent1 -> Parent2 -> NULL head -> L1 -> L2 -> P1 -> P2 -> NULL + * tail -> NULL tail ---------^ + * + * This means that anyone wanting to traverse the whole list to use it just + * needs to start at the head and use the "next" pointers until it goes + * NULL. To add new local content, we insert the item after the tail item + * and update the tail (obviously, if "tail" was NULL, we insert it at the + * head). To clear the local list, WE MUST NOT FREE THE INHERITED CONTENT + * because it is shared between the current list and our parent list(s). + * The easiest way to handle this is to simply truncate the list after the + * tail item and then free the local list from the head. When inheriting + * the list for a new local dir, we just save off the exclude_list_struct + * values (so we can pop back to them later) and set the tail to NULL. + */ + +/* Build an exclude structure given an exclude pattern. The value in "pat" + * is not null-terminated. */ static void make_exclude(struct exclude_list_struct *listp, const char *pat, unsigned int pat_len, unsigned int mflags) { @@ -46,23 +99,45 @@ static void make_exclude(struct exclude_ const char *cp; unsigned int ex_len; + if (verbose > 2) { + rprintf(FINFO, "[%s] add_exclude(%.*s, %s%s)\n", + who_am_i(), (int)pat_len, pat, + mflags & MATCHFLG_MERGE_FILE ? "per-dir-merge" + : mflags & MATCHFLG_INCLUDE ? "include" : "exclude", + listp->debug_type); + } + + if (mflags & MATCHFLG_MERGE_FILE) { + int i; + /* If the local merge file was already mentioned, don't + * add it again. */ + for (i = 0; i < mergelist_cnt; i++) { + struct exclude_struct *ex = mergelist_parents[i]; + if (strlen(ex->pattern) == pat_len + && memcmp(ex->pattern, pat, pat_len) == 0) + return; + } + } + ret = new(struct exclude_struct); if (!ret) out_of_memory("make_exclude"); memset(ret, 0, sizeof ret[0]); - if (exclude_path_prefix) - mflags |= MATCHFLG_ABS_PATH; - if (exclude_path_prefix && *pat == '/') - ex_len = strlen(exclude_path_prefix); - else + if (mflags & MATCHFLG_ABS_PATH) { + if (*pat != '/') { + mflags &= ~MATCHFLG_ABS_PATH; + ex_len = 0; + } else + ex_len = dirbuf_len - module_dirlen - 1; + } else ex_len = 0; ret->pattern = new_array(char, ex_len + pat_len + 1); if (!ret->pattern) out_of_memory("make_exclude"); if (ex_len) - memcpy(ret->pattern, exclude_path_prefix, ex_len); + memcpy(ret->pattern, dirbuf + module_dirlen, ex_len); strlcpy(ret->pattern + ex_len, pat, pat_len + 1); pat_len += ex_len; @@ -81,14 +156,40 @@ static void make_exclude(struct exclude_ mflags |= MATCHFLG_DIRECTORY; } - for (cp = ret->pattern; (cp = strchr(cp, '/')) != NULL; cp++) - ret->slash_cnt++; + if (mflags & MATCHFLG_MERGE_FILE) { + struct exclude_list_struct *lp + = new_array(struct exclude_list_struct, 1); + if (!lp) + out_of_memory("make_exclude"); + lp->head = lp->tail = NULL; + if ((cp = strrchr(ret->pattern, '/')) != NULL) + cp++; + else + cp = ret->pattern; + if (asprintf(&lp->debug_type, " (per-dir %s)", cp) < 0) + out_of_memory("make_exclude"); + ret->u.mergelist = lp; + if (mergelist_cnt == mergelist_size) { + mergelist_size += 5; + mergelist_parents = realloc_array(mergelist_parents, + struct exclude_struct *, + mergelist_size); + if (!mergelist_parents) + out_of_memory("make_exclude"); + } + mergelist_parents[mergelist_cnt++] = ret; + } else { + for (cp = ret->pattern; (cp = strchr(cp, '/')) != NULL; cp++) + ret->u.slash_cnt++; + } ret->match_flags = mflags; - if (!listp->tail) + if (!listp->tail) { + ret->next = listp->head; listp->head = listp->tail = ret; - else { + } else { + ret->next = listp->tail->next; listp->tail->next = ret; listp->tail = ret; } @@ -96,22 +197,263 @@ static void make_exclude(struct exclude_ static void free_exclude(struct exclude_struct *ex) { + if (ex->match_flags & MATCHFLG_MERGE_FILE) { + free(ex->u.mergelist->debug_type); + free(ex->u.mergelist); + mergelist_cnt--; + } free(ex->pattern); free(ex); } -void clear_exclude_list(struct exclude_list_struct *listp) +static void clear_exclude_list(struct exclude_list_struct *listp) { - struct exclude_struct *ent, *next; - - for (ent = listp->head; ent; ent = next) { - next = ent->next; - free_exclude(ent); + if (listp->tail) { + struct exclude_struct *ent, *next; + /* Truncate any inherited items from the local list. */ + listp->tail->next = NULL; + /* Now free everything that is left. */ + for (ent = listp->head; ent; ent = next) { + next = ent->next; + free_exclude(ent); + } } listp->head = listp->tail = NULL; } +/* This returns an expanded (absolute) filename for the merge-file name if + * the name has any slashes in it OR if the parent_dirscan var is True; + * otherwise it returns the original merge_file name. If the len_ptr value + * is non-NULL the merge_file name is limited by the referenced length + * value and will be updated with the length of the resulting name. We + * always return a name that is null terminated, even if the merge_file + * name was not. */ +static char *parse_merge_name(const char *merge_file, unsigned int *len_ptr, + unsigned int prefix_skip) +{ + static char buf[MAXPATHLEN]; + char *fn, tmpbuf[MAXPATHLEN]; + unsigned int fn_len; + + if (!parent_dirscan && *merge_file != '/') { + /* Return the name unchanged it doesn't have any slashes. */ + if (len_ptr) { + const char *p = merge_file + *len_ptr; + while (--p > merge_file && *p != '/') {} + if (p == merge_file) { + strlcpy(buf, merge_file, *len_ptr + 1); + return buf; + } + } else if (strchr(merge_file, '/') == NULL) + return (char *)merge_file; + } + + fn = *merge_file == '/' ? buf : tmpbuf; + if (sanitize_paths) { + const char *r = prefix_skip ? "/" : NULL; + /* null-terminate the name if it isn't already */ + if (len_ptr && merge_file[*len_ptr]) { + char *to = fn == buf ? tmpbuf : buf; + strlcpy(to, merge_file, *len_ptr + 1); + merge_file = to; + } + if (!sanitize_path(fn, merge_file, r, dirbuf_depth)) { + rprintf(FERROR, "merge-file name overflows: %s\n", + merge_file); + return NULL; + } + } else { + strlcpy(fn, merge_file, len_ptr ? *len_ptr + 1 : MAXPATHLEN); + clean_fname(fn, 1); + } + + fn_len = strlen(fn); + if (fn == buf) + goto done; + + if (dirbuf_len + fn_len >= MAXPATHLEN) { + rprintf(FERROR, "merge-file name overflows: %s\n", fn); + return NULL; + } + memcpy(buf, dirbuf + prefix_skip, dirbuf_len - prefix_skip); + memcpy(buf + dirbuf_len - prefix_skip, fn, fn_len + 1); + fn_len = clean_fname(buf, 1); + + done: + if (len_ptr) + *len_ptr = fn_len; + return buf; +} + +/* Sets the dirbuf and dirbuf_len values. */ +void set_excludes_dir(const char *dir, unsigned int dirlen) +{ + unsigned int len; + if (*dir != '/') { + memcpy(dirbuf, curr_dir, curr_dir_len); + dirbuf[curr_dir_len] = '/'; + len = curr_dir_len + 1; + if (len + dirlen >= MAXPATHLEN) + dirlen = 0; + } else + len = 0; + memcpy(dirbuf + len, dir, dirlen); + dirbuf[dirlen + len] = '\0'; + dirbuf_len = clean_fname(dirbuf, 1); + if (dirbuf_len > 1 && dirbuf[dirbuf_len-1] == '.' + && dirbuf[dirbuf_len-2] == '/') + dirbuf_len -= 2; + dirbuf[dirbuf_len++] = '/'; + dirbuf[dirbuf_len] = '\0'; + if (sanitize_paths) + dirbuf_depth = count_dir_elements(dirbuf + module_dirlen); +} + +/* This routine takes a per-dir merge-file entry and finishes its setup. + * If the name has a path portion then we check to see if it refers to a + * parent directory of the first transfer dir. If it does, we scan all the + * dirs from that point through the parent dir of the transfer dir looking + * for the per-dir merge-file in each one. */ +static BOOL setup_merge_file(struct exclude_struct *ex, + struct exclude_list_struct *lp, int flags) +{ + char buf[MAXPATHLEN]; + char *x, *y, *pat = ex->pattern; + unsigned int len; + + if (!(x = parse_merge_name(pat, NULL, 0)) || *x != '/') + return 0; + + y = strrchr(x, '/'); + *y = '\0'; + ex->pattern = strdup(y+1); + if (!*x) + x = "/"; + if (*x == '/') + strlcpy(buf, x, MAXPATHLEN); + else + pathjoin(buf, MAXPATHLEN, dirbuf, x); + + len = clean_fname(buf, 1); + if (len != 1 && len < MAXPATHLEN-1) { + buf[len++] = '/'; + buf[len] = '\0'; + } + /* This ensures that the specified dir is a parent of the transfer. */ + for (x = buf, y = dirbuf; *x && *x == *y; x++, y++) {} + if (*x) + y += strlen(y); /* nope -- skip the scan */ + + parent_dirscan = True; + while (*y) { + char save[MAXPATHLEN]; + strlcpy(save, y, MAXPATHLEN); + *y = '\0'; + dirbuf_len = y - dirbuf; + strlcpy(x, ex->pattern, MAXPATHLEN - (x - buf)); + add_exclude_file(lp, buf, flags | XFLG_ABS_PATH); + if (ex->match_flags & MATCHFLG_NO_INHERIT) + lp->head = NULL; + lp->tail = NULL; + strlcpy(y, save, MAXPATHLEN); + while ((*x++ = *y++) != '/') {} + } + parent_dirscan = False; + free(pat); + return 1; +} + +/* Each time rsync changes to a new directory it call this function to + * handle all the per-dir merge-files. The "dir" value is the current path + * relative to curr_dir (which might not be null-terminated). We copy it + * into dirbuf so that we can easily append a file name on the end. */ +void *push_local_excludes(const char *dir, unsigned int dirlen) +{ + struct exclude_list_struct *ap, *push; + int i; + + set_excludes_dir(dir, dirlen); + + push = new_array(struct exclude_list_struct, mergelist_cnt); + if (!push) + out_of_memory("push_local_excludes"); + + for (i = 0, ap = push; i < mergelist_cnt; i++) { + memcpy(ap++, mergelist_parents[i]->u.mergelist, + sizeof (struct exclude_list_struct)); + } + + /* Note: add_exclude_file() might increase mergelist_cnt, so keep + * this loop separate from the above loop. */ + for (i = 0; i < mergelist_cnt; i++) { + struct exclude_struct *ex = mergelist_parents[i]; + struct exclude_list_struct *lp = ex->u.mergelist; + int flags = 0; + + if (verbose > 2) { + rprintf(FINFO, "[%s] pushing exclude list%s\n", + who_am_i(), lp->debug_type); + } + + lp->tail = NULL; /* Switch any local rules to inherited. */ + if (ex->match_flags & MATCHFLG_NO_INHERIT) + lp->head = NULL; + if (ex->match_flags & MATCHFLG_WORD_SPLIT) + flags |= XFLG_WORD_SPLIT; + if (ex->match_flags & MATCHFLG_NO_PREFIXES) + flags |= XFLG_NO_PREFIXES; + if (ex->match_flags & MATCHFLG_INCLUDE) + flags |= XFLG_DEF_INCLUDE; + else if (ex->match_flags & MATCHFLG_NO_PREFIXES) + flags |= XFLG_DEF_EXCLUDE; + + if (ex->match_flags & MATCHFLG_FINISH_SETUP) { + ex->match_flags &= ~MATCHFLG_FINISH_SETUP; + if (setup_merge_file(ex, lp, flags)) + set_excludes_dir(dir, dirlen); + } + + if (strlcpy(dirbuf + dirbuf_len, ex->pattern, + MAXPATHLEN - dirbuf_len) < MAXPATHLEN - dirbuf_len) + add_exclude_file(lp, dirbuf, flags | XFLG_ABS_PATH); + else { + io_error |= IOERR_GENERAL; + rprintf(FINFO, + "cannot add local excludes in long-named directory %s\n", + full_fname(dirbuf)); + } + dirbuf[dirbuf_len] = '\0'; + } + + return (void*)push; +} + +void pop_local_excludes(void *mem) +{ + struct exclude_list_struct *ap, *pop = (struct exclude_list_struct*)mem; + int i; + + for (i = mergelist_cnt; i-- > 0; ) { + struct exclude_struct *ex = mergelist_parents[i]; + struct exclude_list_struct *lp = ex->u.mergelist; + + if (verbose > 2) { + rprintf(FINFO, "[%s] popping exclude list%s\n", + who_am_i(), lp->debug_type); + } + + clear_exclude_list(lp); + } + + for (i = 0, ap = pop; i < mergelist_cnt; i++) { + memcpy(mergelist_parents[i]->u.mergelist, ap++, + sizeof (struct exclude_list_struct)); + } + + free(pop); +} + static int check_one_exclude(char *name, struct exclude_struct *ex, int name_is_dir) { @@ -125,13 +467,14 @@ static int check_one_exclude(char *name, /* If the pattern does not have any slashes AND it does not have * a "**" (which could match a slash), then we just match the * name portion of the path. */ - if (!ex->slash_cnt && !(ex->match_flags & MATCHFLG_WILD2)) { + if (!ex->u.slash_cnt && !(ex->match_flags & MATCHFLG_WILD2)) { if ((p = strrchr(name,'/')) != NULL) name = p+1; } else if (ex->match_flags & MATCHFLG_ABS_PATH && *name != '/' - && curr_dir[1]) { - pathjoin(full_name, sizeof full_name, curr_dir + 1, name); + && curr_dir_len > module_dirlen + 1) { + pathjoin(full_name, sizeof full_name, + curr_dir + module_dirlen + 1, name); name = full_name; } @@ -148,9 +491,9 @@ static int check_one_exclude(char *name, if (ex->match_flags & MATCHFLG_WILD) { /* A non-anchored match with an infix slash and no "**" * needs to match the last slash_cnt+1 name elements. */ - if (!match_start && ex->slash_cnt + if (!match_start && ex->u.slash_cnt && !(ex->match_flags & MATCHFLG_WILD2)) { - int cnt = ex->slash_cnt + 1; + int cnt = ex->u.slash_cnt + 1; for (p = name + strlen(name) - 1; p >= name; p--) { if (*p == '/' && !--cnt) break; @@ -202,12 +545,11 @@ static void report_exclude_result(char c * case we add it back in here. */ if (verbose >= 2) { - rprintf(FINFO, "[%s] %scluding %s %s because of %spattern %s%s\n", + rprintf(FINFO, "[%s] %scluding %s %s because of pattern %s%s%s\n", who_am_i(), ent->match_flags & MATCHFLG_INCLUDE ? "in" : "ex", - name_is_dir ? "directory" : "file", name, type, - ent->pattern, - ent->match_flags & MATCHFLG_DIRECTORY ? "/" : ""); + name_is_dir ? "directory" : "file", name, ent->pattern, + ent->match_flags & MATCHFLG_DIRECTORY ? "/" : "", type); } } @@ -221,6 +563,13 @@ int check_exclude(struct exclude_list_st struct exclude_struct *ent; for (ent = listp->head; ent; ent = ent->next) { + if (ent->match_flags & MATCHFLG_MERGE_FILE) { + int rc = check_exclude(ent->u.mergelist, name, + name_is_dir); + if (rc) + return rc; + continue; + } if (check_one_exclude(name, ent, name_is_dir)) { report_exclude_result(name, ent, name_is_dir, listp->debug_type); @@ -236,32 +585,102 @@ int check_exclude(struct exclude_list_st * be '\0' terminated, so use the returned length to limit the string. * Also, be sure to add this length to the returned pointer before passing * it back to ask for the next token. This routine parses the "!" (list- - * clearing) token and (if xflags does NOT contain XFLG_WORDS_ONLY) the + * clearing) token and (if xflags does NOT contain XFLG_NO_PREFIXES) the * +/- prefixes for overriding the include/exclude mode. The *flag_ptr * value will also be set to the MATCHFLG_* bits for the current token. */ -static const char *get_exclude_tok(const char *p, unsigned int *len_ptr, - unsigned int *flag_ptr, int xflags) +static const char *get_exclude_tok(const char *p, int xflags, + unsigned int *len_ptr, unsigned int *flag_ptr) { const unsigned char *s = (const unsigned char *)p; unsigned int len, mflags = 0; + int empty_pat_is_OK = 0; if (xflags & XFLG_WORD_SPLIT) { /* Skip over any initial whitespace. */ while (isspace(*s)) s++; - /* Update for "!" check. */ + /* Update to point to real start of rule. */ p = (const char *)s; } + if (!*s) + return NULL; - /* Is this a '+' or '-' followed by a space (not whitespace)? */ - if (!(xflags & XFLG_WORDS_ONLY) + /* Figure out what kind of a filter rule "s" is pointing at. */ + if (!(xflags & (XFLG_DEF_INCLUDE | XFLG_DEF_EXCLUDE))) { + char *mods = ""; + switch (*s) { + case ':': + mflags |= MATCHFLG_PERDIR_MERGE + | MATCHFLG_FINISH_SETUP; + /* FALL THROUGH */ + case '.': + mflags |= MATCHFLG_MERGE_FILE; + mods = "-+Cens"; + break; + case '+': + mflags |= MATCHFLG_INCLUDE; + break; + case '-': + break; + case '!': + mflags |= MATCHFLG_CLEAR_LIST; + mods = NULL; + break; + default: + rprintf(FERROR, "Unknown filter rule: %s\n", p); + exit_cleanup(RERR_SYNTAX); + } + while (mods && *++s && *s != ' ' && *s != '=' && *s != '_') { + if (strchr(mods, *s) == NULL) { + if (xflags & XFLG_WORD_SPLIT && isspace(*s)) { + s--; + break; + } + rprintf(FERROR, + "unknown option '%c' in filter rule: %s\n", + *s, p); + exit_cleanup(RERR_SYNTAX); + } + switch (*s) { + case '-': + mflags |= MATCHFLG_NO_PREFIXES; + break; + case '+': + mflags |= MATCHFLG_NO_PREFIXES + | MATCHFLG_INCLUDE; + break; + case 'C': + empty_pat_is_OK = 1; + mflags |= MATCHFLG_NO_PREFIXES + | MATCHFLG_WORD_SPLIT + | MATCHFLG_NO_INHERIT; + break; + case 'e': + mflags |= MATCHFLG_EXCLUDE_SELF; + break; + case 'n': + mflags |= MATCHFLG_NO_INHERIT; + break; + case 's': + mflags |= MATCHFLG_WORD_SPLIT; + break; + } + } + if (*s) + s++; + } else if (!(xflags & XFLG_NO_PREFIXES) && (*s == '-' || *s == '+') && s[1] == ' ') { if (*s == '+') mflags |= MATCHFLG_INCLUDE; s += 2; - } else if (xflags & XFLG_DEF_INCLUDE) - mflags |= MATCHFLG_INCLUDE; + } else { + if (xflags & XFLG_DEF_INCLUDE) + mflags |= MATCHFLG_INCLUDE; + if (*s == '!') + mflags |= MATCHFLG_CLEAR_LIST; /* Tentative! */ + } + if (xflags & XFLG_DIRECTORY) mflags |= MATCHFLG_DIRECTORY; @@ -274,8 +693,21 @@ static const char *get_exclude_tok(const } else len = strlen(s); - if (*p == '!' && len == 1) - mflags |= MATCHFLG_CLEAR_LIST; + if (mflags & MATCHFLG_CLEAR_LIST) { + if (!(xflags & (XFLG_DEF_INCLUDE | XFLG_DEF_EXCLUDE)) && len) { + rprintf(FERROR, + "'!' rule has trailing characters: %s\n", p); + exit_cleanup(RERR_SYNTAX); + } + if (len > 1) + mflags &= ~MATCHFLG_CLEAR_LIST; + } else if (!len && !empty_pat_is_OK) { + rprintf(FERROR, "unexpected end of filter rule: %s\n", p); + exit_cleanup(RERR_SYNTAX); + } + + if (xflags & XFLG_ABS_PATH) + mflags |= MATCHFLG_ABS_PATH; *len_ptr = len; *flag_ptr = mflags; @@ -287,35 +719,71 @@ void add_exclude(struct exclude_list_str int xflags) { unsigned int pat_len, mflags; - const char *cp; + const char *cp, *p; if (!pattern) return; - cp = pattern; - pat_len = 0; while (1) { - cp = get_exclude_tok(cp + pat_len, &pat_len, &mflags, xflags); - if (!pat_len) + /* Remember that the returned string is NOT '\0' terminated! */ + cp = get_exclude_tok(pattern, xflags, &pat_len, &mflags); + if (!cp) break; + if (pat_len >= MAXPATHLEN) { + rprintf(FERROR, "discarding over-long exclude: %s\n", + cp); + continue; + } + pattern = cp + pat_len; if (mflags & MATCHFLG_CLEAR_LIST) { if (verbose > 2) { rprintf(FINFO, - "[%s] clearing %sexclude list\n", + "[%s] clearing exclude list%s\n", who_am_i(), listp->debug_type); } clear_exclude_list(listp); continue; } - make_exclude(listp, cp, pat_len, mflags); - - if (verbose > 2) { - rprintf(FINFO, "[%s] add_exclude(%.*s, %s%sclude)\n", - who_am_i(), (int)pat_len, cp, listp->debug_type, - mflags & MATCHFLG_INCLUDE ? "in" : "ex"); + if (!pat_len) { + cp = ".cvsignore"; + pat_len = 10; + } + + if (mflags & MATCHFLG_MERGE_FILE) { + unsigned int len = pat_len; + if (mflags & MATCHFLG_EXCLUDE_SELF) { + const char *name = strrchr(cp, '/'); + if (name) + len -= ++name - cp; + else + name = cp; + make_exclude(listp, name, len, 0); + mflags &= ~MATCHFLG_EXCLUDE_SELF; + len = pat_len; + } + if (mflags & MATCHFLG_PERDIR_MERGE) { + if (parent_dirscan) { + if (!(p = parse_merge_name(cp, &len, module_dirlen))) + continue; + make_exclude(listp, p, len, mflags); + continue; + } + } else { + int flgs = XFLG_FATAL_ERRORS; + if (!(p = parse_merge_name(cp, &len, 0))) + continue; + if (mflags & MATCHFLG_INCLUDE) + flgs |= XFLG_DEF_INCLUDE; + else if (mflags & MATCHFLG_NO_PREFIXES) + flgs |= XFLG_DEF_EXCLUDE; + add_exclude_file(listp, p, flgs); + continue; + } } + + make_exclude(listp, cp, pat_len, mflags); } } @@ -324,7 +792,7 @@ void add_exclude_file(struct exclude_lis int xflags) { FILE *fp; - char line[MAXPATHLEN+3]; /* Room for "x " prefix and trailing slash. */ + char line[MAXPATHLEN+11]; /* Room for prefix chars and trailing slash. */ char *eob = line + sizeof line - 1; int word_split = xflags & XFLG_WORD_SPLIT; @@ -338,13 +806,19 @@ void add_exclude_file(struct exclude_lis if (!fp) { if (xflags & XFLG_FATAL_ERRORS) { rsyserr(FERROR, errno, - "failed to open %s file %s", - xflags & XFLG_DEF_INCLUDE ? "include" : "exclude", - fname); + "failed to open %sclude file %s", + xflags & XFLG_DEF_INCLUDE ? "in" : "ex", + safe_fname(fname)); exit_cleanup(RERR_FILEIO); } return; } + dirbuf[dirbuf_len] = '\0'; + + if (verbose > 2) { + rprintf(FINFO, "[%s] add_exclude_file(%s,%d)\n", + who_am_i(), safe_fname(fname), xflags); + } while (1) { char *s = line; @@ -388,7 +862,7 @@ void send_exclude_list(int f) * FIXME: This pattern shows up in the output of * report_exclude_result(), which is not ideal. */ if (list_only && !recurse) - add_exclude(&exclude_list, "/*/*", 0); + add_exclude(&exclude_list, "/*/*", XFLG_DEF_EXCLUDE); for (ent = exclude_list.head; ent; ent = ent->next) { unsigned int l; @@ -402,10 +876,34 @@ void send_exclude_list(int f) p[l] = '\0'; } - if (ent->match_flags & MATCHFLG_INCLUDE) { + if (ent->match_flags & MATCHFLG_PERDIR_MERGE) { + char buf[32], *op = buf; + if (protocol_version < 29) { + rprintf(FERROR, + "remote rsync is too old to understand per-directory filter files.\n"); + exit_cleanup(RERR_SYNTAX); + } + *op++ = ':'; + if (ent->match_flags & MATCHFLG_WORD_SPLIT) + *op++ = 's'; + if (ent->match_flags & MATCHFLG_NO_INHERIT) + *op++ = 'n'; + if (ent->match_flags & MATCHFLG_EXCLUDE_SELF) + *op++ = 'e'; + if (ent->match_flags & MATCHFLG_NO_PREFIXES) { + if (ent->match_flags & MATCHFLG_INCLUDE) + *op++ = '+'; + else + *op++ = '-'; + } + *op++ = ' '; + write_int(f, l + (op - buf)); + write_buf(f, buf, op - buf); + } else if (ent->match_flags & MATCHFLG_INCLUDE) { write_int(f, l + 2); write_buf(f, "+ ", 2); - } else if (*p == '-' || *p == '+') { + } else if (protocol_version >= 29 + || ((*p == '-' || *p == '+') && p[1] == ' ')) { write_int(f, l + 2); write_buf(f, "- ", 2); } else @@ -419,14 +917,15 @@ void send_exclude_list(int f) void recv_exclude_list(int f) { - char line[MAXPATHLEN+3]; /* Room for "x " prefix and trailing slash. */ + char line[MAXPATHLEN+11]; /* Room for prefix and trailing slash. */ + unsigned int xflags = protocol_version >= 29 ? 0 : XFLG_DEF_EXCLUDE; unsigned int l; while ((l = read_int(f)) != 0) { if (l >= sizeof line) overflow("recv_exclude_list"); read_sbuf(f, line, l); - add_exclude(&exclude_list, line, 0); + add_exclude(&exclude_list, line, xflags); } } @@ -443,18 +942,18 @@ static char default_cvsignore[] = void add_cvs_excludes(void) { + static unsigned int cvs_flags = XFLG_WORD_SPLIT | XFLG_NO_PREFIXES + | XFLG_DEF_EXCLUDE; char fname[MAXPATHLEN]; char *p; - add_exclude(&exclude_list, default_cvsignore, - XFLG_WORD_SPLIT | XFLG_WORDS_ONLY); + add_exclude(&exclude_list, ":C", 0); + add_exclude(&exclude_list, default_cvsignore, cvs_flags); if ((p = getenv("HOME")) && pathjoin(fname, sizeof fname, p, ".cvsignore") < sizeof fname) { - add_exclude_file(&exclude_list, fname, - XFLG_WORD_SPLIT | XFLG_WORDS_ONLY); + add_exclude_file(&exclude_list, fname, cvs_flags); } - add_exclude(&exclude_list, getenv("CVSIGNORE"), - XFLG_WORD_SPLIT | XFLG_WORDS_ONLY); + add_exclude(&exclude_list, getenv("CVSIGNORE"), cvs_flags); } --- orig/flist.c 2005-01-01 21:11:00 +++ flist.c 2004-08-12 18:59:28 @@ -40,10 +40,9 @@ extern int module_id; extern int ignore_errors; extern int numeric_ids; -extern int cvs_exclude; - extern int recurse; extern char curr_dir[MAXPATHLEN]; +extern unsigned int curr_dir_len; extern char *files_from; extern int filesfrom_fd; @@ -67,7 +66,6 @@ extern int list_only; extern struct exclude_list_struct exclude_list; extern struct exclude_list_struct server_exclude_list; -extern struct exclude_list_struct local_exclude_list; int io_error; @@ -223,8 +221,6 @@ int link_stat(const char *path, STRUCT_S */ static int check_exclude_file(char *fname, int is_dir, int exclude_level) { - int rc; - #if 0 /* This currently never happens, so avoid a useless compare. */ if (exclude_level == NO_EXCLUDES) return 0; @@ -246,10 +242,7 @@ static int check_exclude_file(char *fnam if (exclude_level != ALL_EXCLUDES) return 0; if (exclude_list.head - && (rc = check_exclude(&exclude_list, fname, is_dir)) != 0) - return rc < 0; - if (local_exclude_list.head - && check_exclude(&local_exclude_list, fname, is_dir) < 0) + && check_exclude(&exclude_list, fname, is_dir) < 0) return 1; return 0; } @@ -983,15 +976,7 @@ void send_file_name(int f, struct file_l if (recursive && S_ISDIR(file->mode) && !(file->flags & FLAG_MOUNT_POINT)) { - struct exclude_list_struct last_list = local_exclude_list; - local_exclude_list.head = local_exclude_list.tail = NULL; send_directory(f, flist, f_name_to(file, fbuf)); - if (verbose > 2) { - rprintf(FINFO, "[%s] popping %sexclude list\n", - who_am_i(), local_exclude_list.debug_type); - } - clear_exclude_list(&local_exclude_list); - local_exclude_list = last_list; } } @@ -1002,6 +987,7 @@ static void send_directory(int f, struct struct dirent *di; char fname[MAXPATHLEN]; unsigned int offset; + void *save_excludes; char *p; d = opendir(dir); @@ -1025,18 +1011,7 @@ static void send_directory(int f, struct offset++; } - if (cvs_exclude) { - if (strlcpy(p, ".cvsignore", MAXPATHLEN - offset) - < MAXPATHLEN - offset) { - add_exclude_file(&local_exclude_list, fname, - XFLG_WORD_SPLIT | XFLG_WORDS_ONLY); - } else { - io_error |= IOERR_GENERAL; - rprintf(FINFO, - "cannot cvs-exclude in long-named directory %s\n", - full_fname(fname)); - } - } + save_excludes = push_local_excludes(fname, offset); for (errno = 0, di = readdir(d); di; errno = 0, di = readdir(d)) { char *dname = d_name(di); @@ -1057,6 +1032,8 @@ static void send_directory(int f, struct rsyserr(FERROR, errno, "readdir(%s)", dir); } + pop_local_excludes(save_excludes); + closedir(d); } @@ -1076,6 +1053,7 @@ struct file_list *send_file_list(int f, char *p, *dir, olddir[sizeof curr_dir]; char lastpath[MAXPATHLEN] = ""; struct file_list *flist; + BOOL need_first_push = True; int64 start_write; int use_ff_fd = 0; @@ -1096,6 +1074,10 @@ struct file_list *send_file_list(int f, exit_cleanup(RERR_FILESELECT); } use_ff_fd = 1; + if (curr_dir_len < MAXPATHLEN - 1) { + push_local_excludes(curr_dir, curr_dir_len); + need_first_push = False; + } } } @@ -1126,6 +1108,15 @@ struct file_list *send_file_list(int f, } } + if (need_first_push) { + if ((p = strrchr(fname, '/')) != NULL) { + if (*++p && strcmp(p, ".") != 0) + push_local_excludes(fname, p - fname); + } else if (strcmp(fname, ".") != 0) + push_local_excludes(fname, 0); + need_first_push = False; + } + if (link_stat(fname, &st, keep_dirlinks) != 0) { if (f != -1) { io_error |= IOERR_GENERAL; --- orig/loadparm.c 2005-01-01 21:11:00 +++ loadparm.c 2005-01-16 19:48:52 @@ -133,6 +133,7 @@ typedef struct char *auth_users; char *secrets_file; BOOL strict_modes; + char *filter; char *exclude; char *exclude_from; char *include; @@ -175,6 +176,7 @@ static service sDefault = NULL, /* auth users */ NULL, /* secrets file */ True, /* strict modes */ + NULL, /* filter */ NULL, /* exclude */ NULL, /* exclude from */ NULL, /* include */ @@ -294,6 +296,7 @@ static struct parm_struct parm_table[] = {"auth users", P_STRING, P_LOCAL, &sDefault.auth_users, NULL, 0}, {"secrets file", P_STRING, P_LOCAL, &sDefault.secrets_file,NULL, 0}, {"strict modes", P_BOOL, P_LOCAL, &sDefault.strict_modes,NULL, 0}, + {"filter", P_STRING, P_LOCAL, &sDefault.filter, NULL, 0}, {"exclude", P_STRING, P_LOCAL, &sDefault.exclude, NULL, 0}, {"exclude from", P_STRING, P_LOCAL, &sDefault.exclude_from,NULL, 0}, {"include", P_STRING, P_LOCAL, &sDefault.include, NULL, 0}, @@ -378,6 +381,7 @@ FN_LOCAL_STRING(lp_hosts_deny, hosts_den FN_LOCAL_STRING(lp_auth_users, auth_users) FN_LOCAL_STRING(lp_secrets_file, secrets_file) FN_LOCAL_BOOL(lp_strict_modes, strict_modes) +FN_LOCAL_STRING(lp_filter, filter) FN_LOCAL_STRING(lp_exclude, exclude) FN_LOCAL_STRING(lp_exclude_from, exclude_from) FN_LOCAL_STRING(lp_include, include) --- orig/options.c 2005-01-17 23:11:45 +++ options.c 2005-01-16 23:34:15 @@ -144,6 +144,7 @@ int list_only = 0; char *batch_name = NULL; static int daemon_opt; /* sets am_daemon after option error-reporting */ +static int F_option_cnt = 0; static int modify_window_set; static char *dest_option = NULL; static char *max_size_arg; @@ -291,6 +292,9 @@ void usage(enum logcode F) rprintf(F," -P equivalent to --partial --progress\n"); rprintf(F," -z, --compress compress file data\n"); rprintf(F," -C, --cvs-exclude auto ignore files in the same way CVS does\n"); + rprintf(F," -f, --filter=RULE add a file-filtering RULE\n"); + rprintf(F," -F same as --filter=': /.rsync-rules'\n"); + rprintf(F," repeated: --filter='- .rsync-rules'\n"); rprintf(F," --exclude=PATTERN exclude files matching PATTERN\n"); rprintf(F," --exclude-from=FILE exclude patterns listed in FILE\n"); rprintf(F," --include=PATTERN don't exclude files matching PATTERN\n"); @@ -320,7 +324,7 @@ void usage(enum logcode F) } enum {OPT_VERSION = 1000, OPT_DAEMON, OPT_SENDER, OPT_EXCLUDE, OPT_EXCLUDE_FROM, - OPT_DELETE_AFTER, OPT_DELETE_EXCLUDED, + OPT_FILTER, OPT_DELETE_AFTER, OPT_DELETE_EXCLUDED, OPT_COMPARE_DEST, OPT_COPY_DEST, OPT_LINK_DEST, OPT_INCLUDE, OPT_INCLUDE_FROM, OPT_MODIFY_WINDOW, OPT_READ_BATCH, OPT_WRITE_BATCH, OPT_TIMEOUT, OPT_MAX_SIZE, @@ -343,6 +347,7 @@ static struct poptOption long_options[] {"delete-excluded", 0, POPT_ARG_NONE, 0, OPT_DELETE_EXCLUDED, 0, 0 }, {"force", 0, POPT_ARG_NONE, &force_delete, 0, 0, 0 }, {"numeric-ids", 0, POPT_ARG_NONE, &numeric_ids, 0, 0, 0 }, + {"filter", 'f', POPT_ARG_STRING, 0, OPT_FILTER, 0, 0 }, {"exclude", 0, POPT_ARG_STRING, 0, OPT_EXCLUDE, 0, 0 }, {"include", 0, POPT_ARG_STRING, 0, OPT_INCLUDE, 0, 0 }, {"exclude-from", 0, POPT_ARG_STRING, 0, OPT_EXCLUDE_FROM, 0, 0 }, @@ -393,6 +398,7 @@ static struct poptOption long_options[] {"ignore-errors", 0, POPT_ARG_NONE, &ignore_errors, 0, 0, 0 }, {"blocking-io", 0, POPT_ARG_VAL, &blocking_io, 1, 0, 0 }, {"no-blocking-io", 0, POPT_ARG_VAL, &blocking_io, 0, 0, 0 }, + {0, 'F', POPT_ARG_NONE, 0, 'F', 0, 0 }, {0, 'P', POPT_ARG_NONE, 0, 'P', 0, 0 }, {"port", 0, POPT_ARG_INT, &rsync_port, 0, 0, 0 }, {"log-format", 0, POPT_ARG_STRING, &log_format, 0, 0, 0 }, @@ -622,10 +628,15 @@ int parse_arguments(int *argc, const cha delete_mode = 1; break; - case OPT_EXCLUDE: + case OPT_FILTER: add_exclude(&exclude_list, poptGetOptArg(pc), 0); break; + case OPT_EXCLUDE: + add_exclude(&exclude_list, poptGetOptArg(pc), + XFLG_DEF_EXCLUDE); + break; + case OPT_INCLUDE: add_exclude(&exclude_list, poptGetOptArg(pc), XFLG_DEF_INCLUDE); @@ -643,8 +654,8 @@ int parse_arguments(int *argc, const cha goto options_rejected; } add_exclude_file(&exclude_list, arg, XFLG_FATAL_ERRORS - | (opt == OPT_INCLUDE_FROM - ? XFLG_DEF_INCLUDE : 0)); + | (opt == OPT_INCLUDE_FROM ? XFLG_DEF_INCLUDE + : XFLG_DEF_EXCLUDE)); break; case 'h': @@ -668,6 +679,19 @@ int parse_arguments(int *argc, const cha am_sender = 1; break; + case 'F': + switch (++F_option_cnt) { + case 1: + add_exclude(&exclude_list, + ": /.rsync-rules", 0); + break; + case 2: + add_exclude(&exclude_list, + "- .rsync-rules", 0); + break; + } + break; + case 'P': do_progress = 1; keep_partial = 1; @@ -966,7 +990,7 @@ int parse_arguments(int *argc, const cha partial_dir = NULL; else if (*partial_dir != '/') { add_exclude(&exclude_list, partial_dir, - XFLG_DIRECTORY); + XFLG_DIRECTORY | XFLG_DEF_EXCLUDE); } keep_partial = 1; } --- orig/rsync.h 2005-01-17 23:11:45 +++ rsync.h 2005-01-17 00:16:04 @@ -109,9 +109,11 @@ #define XFLG_FATAL_ERRORS (1<<0) #define XFLG_DEF_INCLUDE (1<<1) -#define XFLG_WORDS_ONLY (1<<2) +#define XFLG_DEF_EXCLUDE (1<<2) #define XFLG_WORD_SPLIT (1<<3) #define XFLG_DIRECTORY (1<<4) +#define XFLG_NO_PREFIXES (1<<5) +#define XFLG_ABS_PATH (1<<6) #define PERMS_REPORT (1<<0) #define PERMS_SKIP_MTIME (1<<1) @@ -513,11 +515,21 @@ struct map_struct { #define MATCHFLG_INCLUDE (1<<4) /* this is an include, not an exclude */ #define MATCHFLG_DIRECTORY (1<<5) /* this matches only directories */ #define MATCHFLG_CLEAR_LIST (1<<6) /* this item is the "!" token */ +#define MATCHFLG_WORD_SPLIT (1<<7) /* split rules on whitespace */ +#define MATCHFLG_NO_INHERIT (1<<8) /* don't inherit these rules */ +#define MATCHFLG_NO_PREFIXES (1<<9) /* parse no prefixes from patterns */ +#define MATCHFLG_MERGE_FILE (1<<10)/* specifies a file to merge */ +#define MATCHFLG_PERDIR_MERGE (1<<11)/* merge-file is searched per-dir */ +#define MATCHFLG_EXCLUDE_SELF (1<<12)/* merge-file name should be excluded */ +#define MATCHFLG_FINISH_SETUP (1<<13)/* per-dir merge file needs setup */ struct exclude_struct { struct exclude_struct *next; char *pattern; unsigned int match_flags; - int slash_cnt; + union { + int slash_cnt; + struct exclude_list_struct *mergelist; + } u; }; struct exclude_list_struct { --- orig/rsync.yo 2005-01-17 23:11:46 +++ rsync.yo 2005-01-17 07:02:43 @@ -361,6 +361,9 @@ verb( -P equivalent to --partial --progress -z, --compress compress file data -C, --cvs-exclude auto ignore files in the same way CVS does + -f, --filter=RULE add a file-filtering RULE + -F same as --filter=': /.rsync-rules' + repeated: --filter='- .rsync-rules' --exclude=PATTERN exclude files matching PATTERN --exclude-from=FILE exclude patterns listed in FILE --include=PATTERN don't exclude files matching PATTERN @@ -756,14 +759,41 @@ Finally, any file is ignored if it is in .cvsignore file and matches one of the patterns listed therein. See the bf(cvs(1)) manual for more information. -dit(bf(--exclude=PATTERN)) This option allows you to selectively exclude -certain files from the list of files to be transferred. This is most -useful in combination with a recursive transfer. +dit(bf(-f, --filter=RULE)) This option allows you to add rules to selectively +exclude certain files from the list of files to be transferred. This is +most useful in combination with a recursive transfer. -You may use as many --exclude options on the command line as you like +You may use as many --filter options on the command line as you like to build up the list of files to exclude. -See the EXCLUDE PATTERNS section for detailed information on this option. +See the FILTER RULES section for detailed information on this option. + +dit(bf(-F)) The -F option is a shorthand for adding two --filter rules to +your command. The first time it is used is a shorthand for this rule: + +verb( + --filter=': /.rsync-rules' +) + +This tells rsync to look for per-directory .rsync-rules files that have +been sprinkled through the hierarchy and use their rules to filter the +files in the transfer. If -F is repeated, it is a shorthand for this +rule: + +verb( + --filter='- .rsync-rules' +) + +This filters out the .rsync-rules files themselves from the transfer. + +See the FILTER RULES section for detailed information on how these options +work. + +dit(bf(--exclude=PATTERN)) This option is a simplified form of the +--filter option that defaults to an exclude rule and does not allow +the full rule-parsing syntax of normal filter rules. + +See the FILTER RULES section for detailed information on this option. dit(bf(--exclude-from=FILE)) This option is similar to the --exclude option, but instead it adds all exclude patterns listed in the file @@ -771,11 +801,11 @@ FILE to the exclude list. Blank lines i ';' or '#' are ignored. If em(FILE) is bf(-) the list will be read from standard input. -dit(bf(--include=PATTERN)) This option tells rsync to not exclude the -specified pattern of filenames. This is useful as it allows you to -build up quite complex exclude/include rules. +dit(bf(--include=PATTERN)) This option is a simplified form of the +--filter option that defaults to an include rule and does not allow +the full rule-parsing syntax of normal filter rules. -See the EXCLUDE PATTERNS section for detailed information on this option. +See the FILTER RULES section for detailed information on this option. dit(bf(--include-from=FILE)) This specifies a list of include patterns from a file. @@ -820,7 +850,8 @@ was located on the remote "src" host. dit(bf(-0, --from0)) This tells rsync that the filenames it reads from a file are terminated by a null ('\0') character, not a NL, CR, or CR+LF. -This affects --exclude-from, --include-from, and --files-from. +This affects --exclude-from, --include-from, --files-from, and any +merged files specified in a --filter rule. It does not affect --cvs-exclude (since all names read from a .cvsignore file are split on whitespace). @@ -959,8 +990,8 @@ If the partial-dir value is not an absol will prevent partial-dir files from being transferred and also prevent the untimely deletion of partial-dir items on the receiving side. An example: the above --partial-dir option would add an "--exclude=.rsync-partial/" -rule at the end of any other include/exclude rules. Note that if you are -supplying your own include/exclude rules, you may need to manually insert a +rule at the end of any other filter rules. Note that if you are +supplying your own filter rules, you may need to manually insert a rule for this directory exclusion somewhere higher up in the list so that it has a high enough priority to be effective (e.g., if your rules specify a trailing --exclude=* rule, the auto-added rule will be ineffective). @@ -1110,30 +1141,318 @@ page describing the options available fo enddit() -manpagesection(EXCLUDE PATTERNS) +manpagesection(FILTER RULES) -The exclude and include patterns specified to rsync allow for flexible -selection of which files to transfer and which files to skip. - -Rsync builds an ordered list of include/exclude options as specified on -the command line. Rsync checks each file and directory -name against each exclude/include pattern in turn. The first matching -pattern is acted on. If it is an exclude pattern, then that file is -skipped. If it is an include pattern then that filename is not -skipped. If no matching include/exclude pattern is found then the +The filter rules allow for flexible selection of which files to transfer +(include) and which files to skip (exclude). The rules either directly +specify include/exclude patterns or they specify a way to acquire more +include/exclude patterns (e.g. to read them from a file). + +As the list of files/directories to transfer is built, rsync checks each +name to be transferred against the list of include/exclude patterns in +turn, and t first matching pattern is acted on: if it is an exclude +pattern, then that file is skipped; if it is an include pattern then that +filename is not skipped; if no matching pattern is found, then the filename is not skipped. -The filenames matched against the exclude/include patterns are relative -to the "root of the transfer". If you think of the transfer as a -subtree of names that are being sent from sender to receiver, the root -is where the tree starts to be duplicated in the destination directory. -This root governs where patterns that start with a / match (see below). +Rsync builds an ordered list of filter rules as specified on the +command-line. Filter rules have the following syntax: + +itemize( + it() x RULE + it() x=RULE + it() xMODIFIERS RULE + it() xMODIFIERS=RULE + it() ! +) + +The 'x' is a single-letter that specifies the kind of rule to create. It +can have trailing options, and is separated from the RULE by one of the +following characters: a single space, an equal-sign (=), or an underscore +(_). Here are the available rule prefixes: + +verb( + - specifies an exclude pattern. + + specifies an include pattern. + . specifies a merge-file to read for more rules. + : specifies a per-directory merge-file. + ! clears the current include/exclude list +) + +Note that the include/exclude command-line options do not allow the full +range of rule parsing as described above -- they only allow the +specification of include/exclude patterns and the "!" token (not to +mention the comment lines when reading rules from a file). If a pattern +does not begin with "- " (dash, space) or "+ " (plus, space), then the +rule will be interpreted as if "+ " (for an include option) or "- " (for +an exclude option) were prefixed to the string. A --filter option, on +the other hand, must always contain one of the prefixes above. + +Note also that the --filter, --include, and --exclude options take one +rule/pattern each. To add multiple ones, you can repeat the options on +the command-line, use the merge-file syntax of the --filter option, or +the --include-from/--exclude-from options. + +manpagesection(INCLUDE/EXCLUDE PATTERN RULES) + +You can include and exclude files by specifing patterns using the "+" and +"-" filter rules (as introduced in the FILTER RULES section above). These +rules specify a pattern that is matched against the names of the files +that are going to be transferred. These patterns can take several forms: + +itemize( + + it() if the pattern starts with a / then it is anchored to a + particular spot in the hierarchy of files, otherwise it is matched + against the end of the pathname. This is similar to a leading ^ in + regular expressions. + Thus "/foo" would match a file called "foo" at either the "root of the + transfer" (for a global rule) or in the merge-file's directory (for a + per-directory rule). + An unqualified "foo" would match any file or directory named "foo" + anywhere in the tree because the algorithm is applied recursively from + the + top down; it behaves as if each path component gets a turn at being the + end of the file name. Even the unanchored "sub/foo" would match at + any point in the hierarchy where a "foo" was found within a directory + named "sub". See the section on ANCHORING INCLUDE/EXCLUDE PATTERNS for + a full discussion of how to specify a pattern that matches at the root + of the transfer. + + it() if the pattern ends with a / then it will only match a + directory, not a file, link, or device. + + it() if the pattern contains a wildcard character from the set + *?[ then expression matching is applied using the shell filename + matching rules. Otherwise a simple string match is used. + + it() the double asterisk pattern "**" will match slashes while a + single asterisk pattern "*" will stop at slashes. + + it() if the pattern contains a / (not counting a trailing /) or a "**" + then it is matched against the full pathname, including any leading + directories. If the pattern doesn't contain a / or a "**", then it is + matched only against the final component of the filename. Again, + remember that the algorithm is applied recursively so "full filename" can + actually be any portion of a path below the starting directory. + +) + +Note that, when using the --recursive (-r) option (which is implied by +-a), every subcomponent of every path is visited from the top down, so +include/exclude patterns get applied recursively to each subcomponent. +The exclude patterns actually short-circuit the directory traversal stage +when rsync finds the files to send. If a pattern excludes a particular +parent directory, it can render a deeper include pattern ineffectual +because rsync did not descend through that excluded section of the +hierarchy. This is particularly important when using a trailing '*' rule. +For instance, this won't work: + +verb( + + /some/path/this-file-will-not-be-found + + /file-is-included + - * +) + +This fails because the parent directory "some" is excluded by the '*' +rule, so rsync never visits any of the files in the "some" or "some/path" +directories. One solution is to ask for all directories in the hierarchy +to be included by using a single rule: "+_*/" (put it somewhere before the +"-_*" rule). Another solution is to add specific include rules for all +the parent dirs that need to be visited. For instance, this set of rules +works fine: + +verb( + + /some/ + + /some/path/ + + /some/path/this-file-is-found + + /file-also-included + - * +) + +Here are some examples of exclude/include matching: + +itemize( + it() "- *.o" would exclude all filenames matching *.o + it() "- /foo" would exclude a file called foo in the transfer-root directory + it() "- foo/" would exclude any directory called foo + it() "- /foo/*/bar" would exclude any file called bar two + levels below a directory called foo in the transfer-root directory + it() "- /foo/**/bar" would exclude any file called bar two + or more levels below a directory called foo in the transfer-root directory + it() The combination of "+ */", "+ *.c", and "- *" would include all + directories and C source files but nothing else. + it() The combination of "+ foo/", "+ foo/bar.c", and "- *" would include + only the foo directory and foo/bar.c (the foo directory must be + explicitly included or it would be excluded by the "*") +) + +manpagesection(MERGE-FILE FILTER RULES) + +You can merge whole files into your filter rules by specifying either a +"." or a ":" filter rule (as introduced in the FILTER RULES section +above). + +There are two kinds of merged files -- single-instance ('.') and +per-directory (':'). A single-instance merge file is read one time, and +its rules are incorporated into the filter list in the place of the "." +rule. For per-directory merge files, rsync will scan every directory that +it traverses for the named file, merging its contents when the file exists +into the current list of inherited rules. These per-directory rule files +must be created on the sending side because it is the sending side that is +being scanned for the available files to transfer. These rule files may +also need to be transferred to the receiving side if you want them to +affect what files don't get deleted (see PER-DIRECTORY RULES AND DELETE +below). + +Some examples: + +verb( + . /etc/rsync/default.rules + : .per-dir-rules + :n- .non-inherited-per-dir-excludes +) + +The following modifiers are accepted after the "." or ":": + +itemize( + it() A "-" specifies that the file should consist of only exclude + patterns, with no other rule-parsing except for the list-clearing + token ("!"). + + it() A "+" specifies that the file should consist of only include + patterns, with no other rule-parsing except for the list-clearing + token ("!"). + + it() A "C" is a shorthand for the modifiers "sn-", which makes the + parsing compatible with the way CVS parses their exclude files. If no + filename is specified, ".cvsignore" is assumed. + + it() A "e" will exclude the merge-file from the transfer; e.g. + ":e_.rules" is like ":_.rules" and "-_.rules". + + it() An "n" specifies that the rules are not inherited by subdirectories. + + it() An "s" specifies that the rules are split on all whitespace instead + of the normal line-splitting. This also turns off comments. Note: the + space that separates the prefix from the rule is treated specially, so + "- foo + bar" is parsed as two rules (assuming that "-" or "+" was not + specified to turn off the parsing of prefixes). +) + +Per-directory rules are inherited in all subdirectories of the directory +where the merge-file was found unless the 'n' modifier was used. Each +subdirectory's rules are prefixed to the inherited per-directory rules +from its parents, which gives the newest rules a higher priority than the +inherited rules. The entire set of per-dir rules is grouped together in +the spot where the merge-file was specified, so it is possible to override +per-dir rules via a rule that got specified earlier in the list of global +rules. When the list-clearing rule ("!") is read from a per-directory +file, it only clears the inherited rules for the current merge file. + +Another way to prevent a single per-dir rule from being inherited is to +anchor it with a leading slash. Anchored rules in a per-directory +merge-file are relative to the merge-file's directory, so a pattern "/foo" +would only match the file "foo" in the directory where the per-dir filter +file was found. + +Here's an example filter file which you'd specify via --filter=". file": + +verb( + . /home/user/.global-rules + - *.gz + : .rules + + *.[ch] + - *.o +) + +This will merge the contents of the /home/user/.global-rules file at the +start of the list and also turns the ".rules" filename into a per-directory +filter file. All rules read-in prior to the start of the directory scan +follow the global anchoring rules (i.e. a leading slash matches at the root +of the transfer). + +If a per-directory merge-file is specified with a path that is a parent +directory of the first transfer directory, rsync will scan all the parent +dirs from that starting point to the transfer directory for the indicated +per-directory file. For instance, here is a common filter (see -F): + +verb( + --filter=': /.rsync-rules' +) + +That rule tells rsync to scan for the file .rsync-rules in all +directories from the root down through the parent directory of the +transfer prior to the start of the normal directory scan of the file in +the directories that are sent as a part of the transfer. (Note: for an +rsync daemon, the root is always the same as the module's "path".) + +Some examples of this pre-scanning for per-directory files: + +verb( + rsync -avF /src/path/ /dest/dir + rsync -av --filter=': ../../.rsync-rules' /src/path/ /dest/dir + rsync -av --fitler=': .rsync-rules' /src/path/ /dest/dir +) + +The first two commands above will look for ".rsync-rules" in "/" and +"/src" before the normal scan begins looking for the file in "/src/path" +and its subdirectories. The last command avoids the parent-dir scan +and only looks for the ".rsync-rules" files in each directory that is +a part of the transfer. + +If you want to include the contents of a ".cvsignore" in your patterns, +you should use the rule ":C" -- this is a short-hand for the rule +":sn-_.cvsignore", and ensures that the .cvsignore file's contents are +interpreted according to the same parsing rules that CVS uses. You can +use this to affect where the --cvs-exclude (-C) option's inclusion of the +per-directory .cvsignore file gets placed into your rules by putting a +":C" wherever you like in your filter rules. Without this, rsync would +add the per-dir rule for the .cvignore file at the end of all your other +rules (giving it a lower priority than your command-line rules). For +example: + +verb( + cat <"$fromdir/.excl" <"$fromdir/foo/file1" echo removed >"$fromdir/foo/file2" echo cvsout >"$fromdir/foo/file2.old" +cat >"$fromdir/foo/.excl" <"$fromdir/bar/.excl" <"$fromdir/bar/down/to/home-cvs-exclude" +cat >"$fromdir/bar/down/to/.excl2" <"$fromdir/bar/down/to/foo/file1" echo cvsout >"$fromdir/bar/down/to/foo/file1.bak" echo gone >"$fromdir/bar/down/to/foo/file3" echo lost >"$fromdir/bar/down/to/foo/file4" echo cvsout >"$fromdir/bar/down/to/foo/file4.junk" echo smashed >"$fromdir/bar/down/to/foo/to" +cat >"$fromdir/bar/down/to/foo/.excl2" <"$fromdir/mid/.excl2" <"$fromdir/mid/one-in-one-out" echo one-in-one-out >"$fromdir/mid/.cvsignore" echo cvsin >"$fromdir/mid/one-for-all" +cat >"$fromdir/mid/.excl" <"$fromdir/mid/for/one-in-one-out" echo expunged >"$fromdir/mid/for/foo/extra" echo retained >"$fromdir/mid/for/foo/keep" @@ -45,6 +76,7 @@ ln -s too "$fromdir/bar/down/to/foo/sym" excl="$scratchdir/exclude-from" cat >"$excl" <