This blog post is about the current scores of submissions to the Papers track immediately after the completion of reviews.
You will have recently received your first indications of how your submissions might fare in December’s programme committee (PC) meeting. Most papers will have received four reviews. Two reviews have been provided by external reviewers. One of the PC members, the paper’s ‘2AC’ has also provided a review of your submission. The final review is a meta review provided by the paper’s ‘1AC’, another member of the committee.
Across the 2960 papers, 2256 papers (76.2%) received precisely one 1AC review, one 2AC review and two external reviews. Other submissions had a variety of review configurations. For example, 164 submissions received a single 1AC review, as part of the quick/desk reject process. Another 326 papers received three AC reviews, in four cases because the paper only had one external review, but in the other cases because the ACs might have wanted an additional opinion on a paper that split opinion, or a technical aspect of a submission. A total of 409 papers received five or more reviews. Nine papers received six reviews.
A total of 2960 papers were submitted. This year the committee has made more use of the quick/desk reject option available to them. A total of 283 papers (9.6%) have already been rejected. This is a substantial increase from 55 last year. They were rejected because:
- They broke this year’s length restrictions (24)
- They broke the CHI’s anonymity policy (71)
- They were considered out of scope for the conference (8)
- They did not meet the conference’s (very relaxed this year) formatting requirements (1)
- In the opinion of the committee they did not meet the quality threshold to go out for review (179)
The lowest score a reviewer can assign a submission is one, and five is the highest. Reviewers also self-report their expertise as it relates to a given submission. This is rated on a scale from one (least knowledgeable) to four (most knowledgeable). Together, these scores give you an idea of how your paper has fared.
Each submission has an ‘Overall Score’, which is the mean of the scores given by the ACs and reviewers of a submission. The mean of these overall scores is currently 2.48 (SD=0.76). If your paper has an overall score of 3.5 or better, it is currently in the top 10% of submissions by score. Only 105 papers scored a mean of 4.0 or better (3.5%). The distribution of scores is illustrated in Figure 1. Overall, scores are lower on average this year, with this year’s mean of 2.48 lower than the mean score of 2.56 at the same stage last year. In 2016 the mean score at the same point was 2.63.
In other words, overall reviewers lean toward rejecting most papers, with around a quarter of papers receiving a neutral or positive response from reviewers. Experienced submitters to CHI will recognise the distribution of scores. It is quite similar each year. If this is your first time submitting to CHI, then the score distribution should help you contextualize the ratings your submissions have received. The main takeaway is that almost half (1473, 49.8%) of all submissions score ≥2.0 and <3.0.
A huge amount of effort goes into reviews. The latest data we have shows there are 11,175 reviews in the system. These reviews comprise 6,137,208 words. The longest was a whopping 4550 words. The shortest was 34 words. There were 969 reviews of over 1000 words, and the mean length was 549 words (SD=332). The distribution of review length is illustrated in Figure 2.
We also had a look at whether there was a relationship between overall score (i.e., mean score) and mean review length. (We use means because an individual only has nine score choices and the plot is a little dull, then.) We plot this relationship in Figure 3. Note that submissions with very high scores and submissions with very low scores tend to receive the shortest reviews, on average.
One of the least visible parts of the review process to authors is the discussions that reviewers have about submissions in PCS. Before rebuttals were sent out, submissions had received between zero and 25 discussion comments with a median of two. One might expect that as reviewers diverge more in their scoring of a paper, so the amount of discussion they have increases. Figure 4 suggests that this is indeed the case.
You have a short window (see the submission page for specifics about timing) to submit your rebuttal. There is no requirement that you submit a rebuttal, although reviewers and ACs are often pleased to receive some kind of acknowledgement of the time and effort they have spent on your work.
People often wonder whether it is ‘worth’ submitting a rebuttal at or below a certain score. Last year, the overall mean for scores did not move after rebuttals, but this does not mean that individual scores do not move. They can and they do. If the ACs for one of your submissions have encouraged you to write a rebuttal then it might make sense to do so. If nothing else, a rebuttal gives you the chance to consider the perceived limitations of your submission so that you can fix them, whether for CHI 2019 or another venue.
A number of members of our community have provided views on how to write a rebuttal. Here are a few that were suggested in last year’s blog:
- Writing rebuttals by Niklas Elmqvist, University of Maryland, College Park
- Writing CHI Rebuttals by Gene Golovchinsky
- SIGCHI Rebuttals: Some Suggestions How to Write Them by Albrecht Schmidt
- A CHI Rebuttal by Simone O’Callaghan
- How to Write SIGCHI Rebuttals by Hyunyoung Song
This list isn’t exhaustive. There is other great advice floating around, often tagged with the CHI 2019 hashtag.
