Only call add_uid() or add_gid() if numeric_ids is not set.