Only rename a directory with the backup suffix if we're removing