Faster, better, stronger: The case for one-phase reviews

When running a scientific conference, one has to decide which submitted papers get published (and presented), and which ones are not.  Each paper gets a number of reviews; and based on these, the program committee decides which papers get accepted.

Flooded with submissions, several SE conferences went for a two-phase model to reduce workload: A paper first gets two reviews, and only if at least one of these favors acceptance, the paper gets a third review.  As PC chairs of ASE 2013, we went back to a single-phase model, though.  Several people have asked us why this was so.  The answer is: time.  With a one-phase review, you can cut the time required for reviews in half, thus allowing for more, better, and more recent papers.


The math is as follows. Let's assume your conference gets 314 papers submitted (as we had for ASE):

• In a two-phase model,
•  314 papers would have gotten two reviews, totaling 618 reviews.
• 1/3 of papers are eliminated in the first round, leaving 210 which get a third review.
• Total number of reviews: 618 + 210 = 828.
• In a one-phase model,
•  314 papers get three reviews, totaling 942.

By having the one-phase model, your conference will thus have 14% more reviews – in our case, 22–23 papers to review per PC member instead of 20.  However, going for two phases requires a lot more administrative overhead – for the PC chairs, of course, but also for the reviewers.  In particular, you need to allocate two phases in which to do the reviews – instead of, say, eight weeks, where reviewers would be free to allocate their load, you now have six + four weeks, each of which may be more easily blocked by travel, holidays, etc.  Plus, as a PC chair, you need at least another week in between to reallocate and inform the new reviewers.

All in all, you thus trade 14% more reviews against an extension of the review period by three weeks.  Not necessarily a good thing.  In addition, this assumes that papers are all equal; but they’re not: The 14% extra papers are mostly papers that would be quick rejections anyway.  So, not much damage either.

Assuming that more than a third of papers may be rejected in the first round favors the two-phase model.  But then, there’s the concern that by having only two reviewers, you run a greater risk of one reviewer being incompetent, unwilling, lazy, etc.  (It happens to anyone.)  So you may be have to introduce rebuttals to compensate for potential misjudgments, and here go another two weeks of reviewing period.  And then, someone will have to read these rebuttals, which adds more overhead.

To illustrate how two-phase reviews and rebuttals eat up time in contrast to one-phase reviews, let's take a look at ICSE 2014, whose papers are due September 13, 2013.  Notification is January 17 – that is, more than four months later.   For ASE (one phase, no rebuttals), this was May 17 to July 25 – two months and one week, or roughly half the time ICSE takes.

With its short reviewing period, ASE 2013 had more than three times the number of submissions compared to last year, which gave us a great choice of papers.  ICSE will also have a record number of submissions; but if you miss the deadline, chances are you will find another conference which will publish your paper well before ICSE.

Summary: Get rid of frills, and allow authors six more weeks to prepare their submissions.  You will get more, better, and more recent papers – and more, better, independent reviews at the same time.