Make the evaluator's review weights configurable.

[match/match.git] / notes

For funding agencies such as NSF program directors 
that co-ordinate panels, assigning proposals to 
reviewers is a major challenge. It is important that each proposal be
reviewed by qualified experts, and at the same time we would like the
workload across different reviewers to be roughly balanced. The same
issue arises for a program committee chair, who may have to assign
literally hundreds of papers to a program committee consisting of
thirty to forty program committee members. 

What does CMT use? What does Easychair use?

From now on we will focus on the problem of assigning papers to
reviewers. 
We assume that each reviewer is given access to the
list of papers to be reviewed, and provides input on their
preferences by giving a ``desirability'' score to each paper.
We also assume that each paper  has to be reviewed by $at least r$
reviewers. 

List of codes.

We do not consider stable marriage type preference lists,
because a strict ranking of papers would be rather tedious
to produce. In this scheme, the papers are essentially grouped
into a few categories.


Ideally, from the perspective of the papers, we would like to
assign each paper the $r$ ``best'' reviewers for the paper.
Ofcourse, this would lead to a load imbalanced solution where
the load on some program committee members is very high, and the
load on others is low. On the other hand, we could insist 
on a perfectly load balanced solution in which the number
of papers assigned to each program committee member do not
exceed $\lceil rN/P  \rceil$, where $N$ is the number of
submissions and $P$ is the number of program committee members.
Recall that each paper needs $r$ reviewers, so a total of $rN$
reviews need to be generated. However this may lead to a solution
which is not optimal from the perspective of the papers.

One of our goals is to study precisely this tradeoff, and allow each
reviewer to have upto $\lceil rN/P  \rceil + C$ papers assigned to
them, where $C$ is the {\em imbalance factor}. The main question
we consider is: is it possible to obtain a high quality
assignment with a fairly low value of $C$?

One can ask the same question from the perspective of the reviewers
as well. Each reviewer would ideally like papers
that are the ``most desirable'' from their point of view.

{\em Stinkers} are papers that pretty much no-one wanted to review. 
We would like to spread the load of the stinkers as evenly as possible.