Alt.chi is a juried track, with an open review period until 21st January 2019. Anyone can register to review alt.chi papers, and we encourage you to explore the submissions and contribute to discussions. Reviews and discussions will be used in final jury decisions. Notifications will be sent on 30th January, and Camera Ready is 8th February for accepted papers.
PCS, the system used by CHI, doesn’t make the open review process easy, but we have come up with some step-by-step instructions on how to review on the alt.chi page here:
- Although this is an open review process, there are a few steps necessary for New PCS to allow access to the submissions:
Please volunteer to review if you haven’t already. To do this, visit the volunteer page and select SIGCHI CHI2019, and enter a number of reviews for alt.chi.
- You will then have access to the “bidding” page. You will see a list of all available titles and abstracts. For any you wish to review, please select “want” or “willing”. You can bid on more papers at any time during the review period.
- The chairs will assign you every paper you bid on, unless particular papers are oversubscribed. This is not automatic but is done regularly. You will receive an email confirmation when a paper is assigned.
- Once assigned, you will have access to the submission, and any existing discussion and reviews through your reviewing page.
The deadline for reviews is 21st January 2019. However, as reviews are open, we encourage you to submit as soon as possible, to allow for discussion among the other community members. You may revise your review until 21st January.
There are a lot of submissions, and a short amount of time to collect reviews, so please do share the link to the review instructions (above) among your networks. Of course, we hope that you also spend some time reviewing other submissions.
There looks to be a lot of exciting work this year and we can’t wait to see what everyone thinks!
Conor, Ann & Ben
Dear community members,
We recently selected Student Volunteers for CHI 2019. As of now we have 175 lucky students that will get to be SVs at CHI and help us make sure that the conference runs well and smoothly. This represents an acceptance rate of only about 15% as we had 1,204 students enroll to become SVs. While the SV program generally relies on official CHI media posts and word-of-mouth to get students enrolled in the selection process, this year as a way to try to improve the diversity in our lottery pool we have also reached out to all local SIGCHI chapters asking them to share with their students that enrollment was open. The final selection of students is partially random and partially otherwise selected. Let’s break down what that means and the numbers. There are three ways for a student to become a student volunteer at CHI:
PC Meeting SVs
These are SVs that are local to the city where the Program Committee (PC) meeting will happen. These students are locally recruited and selected based on their availability in relation to the required hours for the PC meeting. This year 19 students were PC meeting SVs (~11%).
Students can be nominated by members of the Program or Organizing Committees to become SVs. This year 40 slots (~23%) were reserved for these students. We received 227 nominations, but unfortunately 29 of those students did not register at chisv.org, so they were not considered. This means that there were 198 students for 40 slots, an acceptance rate of about 25%. The SVs selected from the nominated students were selected in one of two ways:
- 15 of the 40 slots were selected based on the information provided by the nominator. We looked for strong recommendations on the person’s ability to perform SV-related tasks, we looked for opportunities to increase the diversity of the SV group, and we looked for people who would benefit the most from being an SV for the current year.
- 25 of the 40 slots were selected through a random lottery.
General Selection SVs
Students can also be selected from the general selection process to become SVs. This year there were 106 slots available and, at the time of the selection, 1,141 students enrolled. This means only a 9% chance of getting in! There are two ways for a student to be selected at this step:
- 35 of 106 slots (or 20% of total slots) were selected as “institutional knowledge SVs.” These are students that have been SVs at CHI before, are experienced with a variety of tasks, and can help train the incoming class of SVs. All of them were exceptional SVs in previous years (e.g., always on time AND very proactive AND helpful to others on/off duty AND went above the requirements for their current task). Many of them are trained in specialized tasks. Unfortunately due to the high competition for a SV slot, we couldn’t accept all students that fit this description. There were many more past SVs we would have loved to have back, but it was important to us to give people who are new to CHI a chance as well. We could only bring back approximately 50% of past SVs that were in this group.
- The remaining 81 of the 106 slots (or ~46% of the total slots) are randomly selected from a weighted lottery.
The current lottery system, built into chisv.org, assigns one ticket per student enrolled. We can also increase the number of tickets assigned to a student (a.k.a. influence the likelihood that a type of student has to win the lottery) based on:
- Whether a student is local;
- Whether a student has attended CHI before;
- Whether a student needs a VISA for the conference;
- The type of program the student is currently in (BA/MA/PhD); and,
- Whether a student has been a SV at CHI before.
This year, to ensure we gave opportunities to many new students, we granted two extra tickets to people who have NOT attended CHI before and one extra ticket to students who have NOT been SVs at CHI before. Thus, the likelihood to get accepted increases for people with higher ticket counts (up to 4 this year), but students with low ticket counts still have a chance to win the lottery. As all of the factors mentioned above are optional, we did not give extra tickets depending on whether the student is local, whether they need a visa, or their level of study, as we did not believe these aspects aligned with our goals.
Lastly, we are working with a student to upgrade the chisv.org system. This means improving parts related to logistics during the conference, making the interface more inclusive and accessible, and improving the lottery process. We are investigating opportunities for future chairs to tailor the population of SVs, especially to increase the diversity of the SV program. If you have suggestions on aspects that should be built into the new lottery system, please consider taking a few minutes and writing up your opinions here.
And who got in?
While these numbers are not final as sometimes students give up their SV slot (e.g. because of visas, funding, personal reasons, etc.), they represent the SVs that were selected from the process above. The CHI 2019 SVs are:
- from 39 different nationalities and 100 different universities;
- almost half first time CHI attendees (85, 49%);
- largely new to the CHI SV program (126, ~72% have never been CHI SVs);
However, there are two big things to note. All information is self-reported and we rely on students filling them correctly. Also, our incoming SVs are only as diverse as the pool from which the lottery is drawn! We worked hard this year to reach out to all local SIGCHI chapters, but the best way to ensure more diversity in the SV program is to encourage students in your country, university, and department to enroll.
Student volunteers have become an essential part in the organization of CHI. They play a major role in executing structural tasks – especially during the conference. Among other things: we stuff your conference bags; hand out and check badges; show you where to find a paper session, restaurant, bathroom, your lost water bottle, etc; help set up nets for drones, build sculptures out of coke bottles and other exciting demos; also help figure out where is that missing paper presenter and why, oh why, isn’t the microphone working anymore? Along with many others, the student volunteers put A LOT of effort into helping CHI run smoothly.
SVs are also HCI researchers. Quite a few SVs already publish their research at CHI and have been attending conferences for a while. For others, CHI is a whole new experience, allowing them to see how research results are distributed and how the community interacts. In both cases, being an SV is an incredible opportunity to network with possible mentors, collaborators, and peers.
And here is the tricky bit: SVs are students! We are not trained event managers or AV technicians. We are volunteers and conference attendees. All SVs agree to an informal contract: in exchange for 20 hours of their time (many put in much more), the conference waives their registration fees and provides daily lunch and breakfast (a lot of tasks start as early as 6:30am, some go well through lunch, and others end as late as 8pm). SVs still have to pay for their own housing and transportation. Workshops and courses are not covered either.
In the scope of their 20 hours, SVs engage in plenty of tasks some of which are simple, while others require certain expertise or training. For this reason we need some people who have done the job before and can teach the job to the next generation of SVs. This is the basic concept: returning SVs show incoming SVs how tasks work, new students come up with new ideas on how to improve them.
And who are the SV chairs?
SV chairs are two senior SVs who have seen the process through multiple years of serving as SVs. The SV chair position is a two year commitment (one junior chair and one senior chair). This is because the CHI SV program is a beast. With 175 SVs each year, coordination with many conference chairs, multiple stakeholders with different needs, serving as SV chair for CHI requires SV experience and training in the role. The first year the junior SV chair observes, learns, and helps with organizational tasks. They learn how to operate chisv.org, how to address different types of requests, and how to manage an operation the size of a small startup.
Future SV chairs are selected based on their experience as SVs, their graduation timeline, and the specific needs of the conference that year. The current SV chairs consider several candidates and make a recommendation to the General Chairs of someone they are confident will do a good job in organizing the SV program for the coming years. Sometimes, when there are special needs the General Chairs of future CHIs are brought in earlier in this decision process, to ensure that the “rollover” SV Chair will be able to attend to those needs (e.g. locations where English is not as widely spoken or where cultural norms are significantly different).
As SV chairs, we invest more than 100 pre-conference hours planning SV tasks and ensuring the conference is adequately supported by the SV program. During the conference we spend the majority the conference managing and addressing incoming requests. If you would like to be a future SV chair, make sure you are a SV at CHI (and other conferences as well), have higher responsibilities tasks (volunteer for them!), and that you let us know that you are interested. Every year we go through the process of picking a new junior chair and that person can be you!
In this blog we consider the big picture from the CHI 2019 Papers track. We look at what’s been accepted, what hasn’t. We also consider how things went with reviewing and look at some author-level data. We will analyze the effect of rebuttals in this cycle in another blog post.
This year, 2958 papers were left in the system for review. Of these, 705 have been provisionally accepted for the Papers track at CHI 2019. That is a 23.8% acceptance rate, down from 25.7% last year. Eighty-three of these accepts are being shepherded*. The overall mean score for all submissions, including desk and quick rejects, was 2.52 (SD=0.35).
As in previous years, a significant proportion of papers score between 2.0 and 3.0 (inclusive) – this year more than half (1,494 submissions, 50.5%). Figure 1 shows the distribution of accepts by score. The figure collapses accepted and shepherded papers into ‘A’s and all other decisions (including withdrawal) into ‘R’s. Accepted papers had a mean score of 3.66 (Range 2.38-4.88). This was lower than last year’s mean of 3.73. Forty-two papers with mean scores lower than 3.0 have been accepted (which is precisely double last year’s 21).
The highest-scoring reject had an average of 3.62. There were 117 submissions rejected with a score of ≥3.0 (but only 65 >3.0). There were 224 submissions that scored >2.8 and <3.2. Of these, 70 were accepted. Submissions averaging >3.25 (n=587) were extremely likely to be accepted (97.1%).
A total of 2799 external reviewers completed 5433 external reviews (Mdn=1, M=1.94). As you will see in the next section, this means that authors outnumber reviewers just under 3:1. The General Chairs, TPCs and Papers Chairs thank reviewers for this absolutely essential contribution. No reviews means no accepts, no accepts means no conference.
The most reviews completed by an external reviewer was 12. Five or more reviews were completed by 189 reviewers. The distribution of completed reviews by a given reviewer is illustrated in Figure 2.
External reviewers are only half the picture, though. The other half is the extremely dedicated programme committee, whose associate chairs write reviews and meta reviews and who made the decisions over the 7th and 8th of December. The 400 members of the PC produced 6074 reviews (or around 15 each). Just producing the reviews is a gargantuan amount of work. Unfortunately we don’t have a chart that shows the work that goes into finding external reviewers, managing those reviewers, sending reminders, facilitating discussions, attending the PC meeting, writing final post-PC comments, shepherding, checking camera-ready submissions,… it goes on and on. The General Chairs, TPCs and Papers Chairs are extremely grateful to the ACs for their efforts in putting together CHI’s biggest ever programme.
A total of 8,319 authors were listed on submissions, give or take a few duplicate accounts. The vast majority of authors (6,425, 77%) were involved in one submission. Of these authors, 1,462 (22.8%) had their names on an accepted submission. Being on a large number of submissions is uncommon. Only 265 of those 8,319 authors (3%) made five or more submissions. Only 13 authors (0.15%) have their names attached to five or more accepted papers. We congratulate these authors on their success, but note that the vast majority of authors submit one paper and that submission is not successful. Do not be disheartened if things have not gone as well as you might have hoped. You might consider submitting a revised version of the paper to a different SIGCHI conference, many of which will be issuing calls of submissions soon.
Are people who make more submissions generally more successful with them? Only thirteen authors had three or more accepts without a reject. Two authors had four. Conversely, 45 authors made five or more submissions without any success. Figure 4 helps to make this a little bit clearer. It plots, for each author, how many accepts they had, versus the number of rejects they had. Note, authors with more than four accepts (n=13) are excluded from this plot as the published proceedings may allow for their identity to be determined. Authors with more than 7 rejects (n=25) are also excluded.
*ACs can require changes to papers before the camera-ready submissions are made. A small number of papers are assigned a ‘shepherd’. This is an AC who helps the authors make changes to the manuscript in a way that meets the requirements of the PC and reviewers. It often happens when a paper needs specific changes that are carefully made in order for it to be it accepted.
This blog post is about the current scores of submissions to the Papers track immediately after the completion of reviews.
You will have recently received your first indications of how your submissions might fare in December’s programme committee (PC) meeting. Most papers will have received four reviews. Two reviews have been provided by external reviewers. One of the PC members, the paper’s ‘2AC’ has also provided a review of your submission. The final review is a meta review provided by the paper’s ‘1AC’, another member of the committee.
Across the 2960 papers, 2256 papers (76.2%) received precisely one 1AC review, one 2AC review and two external reviews. Other submissions had a variety of review configurations. For example, 164 submissions received a single 1AC review, as part of the quick/desk reject process. Another 326 papers received three AC reviews, in four cases because the paper only had one external review, but in the other cases because the ACs might have wanted an additional opinion on a paper that split opinion, or a technical aspect of a submission. A total of 409 papers received five or more reviews. Nine papers received six reviews.
A total of 2960 papers were submitted. This year the committee has made more use of the quick/desk reject option available to them. A total of 283 papers (9.6%) have already been rejected. This is a substantial increase from 55 last year. They were rejected because:
- They broke this year’s length restrictions (24)
- They broke the CHI’s anonymity policy (71)
- They were considered out of scope for the conference (8)
- They did not meet the conference’s (very relaxed this year) formatting requirements (1)
- In the opinion of the committee they did not meet the quality threshold to go out for review (179)
The lowest score a reviewer can assign a submission is one, and five is the highest. Reviewers also self-report their expertise as it relates to a given submission. This is rated on a scale from one (least knowledgeable) to four (most knowledgeable). Together, these scores give you an idea of how your paper has fared.
Each submission has an ‘Overall Score’, which is the mean of the scores given by the ACs and reviewers of a submission. The mean of these overall scores is currently 2.48 (SD=0.76). If your paper has an overall score of 3.5 or better, it is currently in the top 10% of submissions by score. Only 105 papers scored a mean of 4.0 or better (3.5%). The distribution of scores is illustrated in Figure 1. Overall, scores are lower on average this year, with this year’s mean of 2.48 lower than the mean score of 2.56 at the same stage last year. In 2016 the mean score at the same point was 2.63.
In other words, overall reviewers lean toward rejecting most papers, with around a quarter of papers receiving a neutral or positive response from reviewers. Experienced submitters to CHI will recognise the distribution of scores. It is quite similar each year. If this is your first time submitting to CHI, then the score distribution should help you contextualize the ratings your submissions have received. The main takeaway is that almost half (1473, 49.8%) of all submissions score ≥2.0 and <3.0.
A huge amount of effort goes into reviews. The latest data we have shows there are 11,175 reviews in the system. These reviews comprise 6,137,208 words. The longest was a whopping 4550 words. The shortest was 34 words. There were 969 reviews of over 1000 words, and the mean length was 549 words (SD=332). The distribution of review length is illustrated in Figure 2.
We also had a look at whether there was a relationship between overall score (i.e., mean score) and mean review length. (We use means because an individual only has nine score choices and the plot is a little dull, then.) We plot this relationship in Figure 3. Note that submissions with very high scores and submissions with very low scores tend to receive the shortest reviews, on average.
One of the least visible parts of the review process to authors is the discussions that reviewers have about submissions in PCS. Before rebuttals were sent out, submissions had received between zero and 25 discussion comments with a median of two. One might expect that as reviewers diverge more in their scoring of a paper, so the amount of discussion they have increases. Figure 4 suggests that this is indeed the case.
You have a short window (see the submission page for specifics about timing) to submit your rebuttal. There is no requirement that you submit a rebuttal, although reviewers and ACs are often pleased to receive some kind of acknowledgement of the time and effort they have spent on your work.
People often wonder whether it is ‘worth’ submitting a rebuttal at or below a certain score. Last year, the overall mean for scores did not move after rebuttals, but this does not mean that individual scores do not move. They can and they do. If the ACs for one of your submissions have encouraged you to write a rebuttal then it might make sense to do so. If nothing else, a rebuttal gives you the chance to consider the perceived limitations of your submission so that you can fix them, whether for CHI 2019 or another venue.
A number of members of our community have provided views on how to write a rebuttal. Here are a few that were suggested in last year’s blog:
- Writing rebuttals by Niklas Elmqvist, University of Maryland, College Park
- Writing CHI Rebuttals by Gene Golovchinsky
- SIGCHI Rebuttals: Some Suggestions How to Write Them by Albrecht Schmidt
- A CHI Rebuttal by Simone O’Callaghan
- How to Write SIGCHI Rebuttals by Hyunyoung Song
This list isn’t exhaustive. There is other great advice floating around, often tagged with the CHI 2019 hashtag.
The CHI 2019 Papers track, which is bigger by submission volume and organizational effort than the rest of the CHI tracks combined, closed last Friday. In this blog post we look at what has happened so far with submissions and to give you an idea of what will happen next and to encourage you to be part of the ‘chi circle of kindness’ by contributing to the community review effort.
On the 14th of September, 3784 abstracts were submitted to PCS, the conference’s submission system. By the deadline on the 21st of September, there were 2966 complete submissions left in PCS. This means that 22% of submissions at the abstract phase were either deleted or left incomplete by the deadline.
Excepting any submissions that are rejected early in the process, these 2966 papers (a 14% increase in complete paper submissions compared to CHI2018) will each receive four reviews: two external reviews, one internal review from the 2AC and one meta-review from the 1AC. Therefore, the submissions will require almost 12,000 reviews. This is a huge amount of work.
We need your help as part of the community to get through this work. Last year 2651 reviewers produced 5105 reviews. But most people only wrote one. This meant that 1053 reviewers produced 3621 reviews. Remember that this year we will need additional external reviews to ensure everyone’s submissions get the kind of attention that makes CHI a high-quality venue for publishing.
So we’d love you to volunteer to review here, especially so if you are an author as each paper you have submitted generates four reviews: https://new.precisionconference.com/review_volunteering
We’re doing all of this with the added ‘excitement’ of using PCS 2.0 for the first time for CHI – please be patient as we deal with the inevitable teething problems that come with using a new system. And dealing with teething problems also takes a huge additional effort over and above the huge volunteer effort already involved in making CHI happen. For this we already owe a huge debt of thanks to the papers chairs (and many other members of the committee and the SIGCHI EC) who have been working on our behalf to make it all happen!
TPC chairs for CHI 2019, Anna Cox and Vassilis Kostakos
Analytics chair for CHI 2019, Sandy Gould
We’re making some changes to the papers reviewing and submission process this year, and thought it would be useful to communicate this and the reasons for these changes to the community. There are a few differences to the ways that we’ll be handling things this year that we want to highlight – use of an all-virtual meeting, use of quick rejects, use of a new version of PCS, and an update to the paper templates. We’ll document these changes below, and we feel strongly that these are necessary for sustaining the conference over the long term, ensuring the quality of the research presented, providing an equitable playing field for authors, and helping us – the CHI community – to work more effectively.
This year, the PC meeting held in December will be almost all-virtual. The general chairs, technical program chairs, paper chairs, subcommittee chairs and some student volunteers will be present at a meeting in Glasgow. But, ALL the Associate Chairs (ACs) will be virtual. In past years, we have experimented with virtual subcommittees, starting with one subcommittee for CHI 2016, three for CHI 2017 and six for CHI 2018. This year, all 12 subcommittees will be running virtually. The ACs will all be remote, and participating electronically in a synchronously held PC meeting. The decision to go completely virtual was not taken lightly, and was approved by the CHI Steering Committee as a trial for this year. The decision was based on many factors: while the value of in-person (non-virtual meetings) is high, there are a number of reasons that a virtual PC meeting makes sense:
- the cost of running such a large meeting (larger than almost all SIGCHI conferences!)
- the environmental sustainability of hundreds of people flying to the PC meeting and then again to the conference (plus the travel time itself)
- and the desire to grow our community by recruiting ACs that otherwise couldn’t participate in an in-person meeting due to cost of travel, health, family, etc.
As we have challenges with time zones (not all ACs can participate synchronously for the full day meeting in Glasgow – for example, for ACs on the west coast of the United States and Canada, the PC meeting starts at 1am and ends at 9am each day, for 2 days), we are not asking all ACs to attend the entire PC meeting. Instead, based on their time zone, ACs will be asked to participate in the PC meeting for a particular window of time (minimum of 5 hours each day).
As well, we are encouraging ACs to participate more in the discussion of papers where they were not assigned a 1AC/2AC/3AC role. We are formalizing this role as observer, and are hoping that just like in a physical PC meeting, observers will participate in the discussion of a paper at the PC meeting. This may not be possible depending on time zone availability, so we are also asking observers to add to the PCS discussion forum for papers so their input can be considered at the PC meeting even if they are not present to participate in the synchronous discussion.
As Papers Chairs, we will work with SC’s to monitor how discussions of papers are managed at the PC meeting to ensure that papers are handled fairly and equitably.
Use of Quick Rejects
In the papers review process for CHI 2018, we had papers that were rejected early in the process, before papers went out for review. Some were desk rejected (out of scope, over the page limit, or some other egregious issue) and some were quick rejected (missing something critical that would make replication, analysis or validation of claims impossible, insufficient contribution, etc.). In both cases, the papers were looked at by 2 ACs, the Subcommittee Chairs, and the Papers Chairs, but did not go out for external review, to save reviewer effort. However, these papers only accounted for 1.7% of the submitted papers. With expectations that the conference submission rate will continue to grow at ~8% per year, and to focus the community’s reviewing energy on the papers that are more likely to be accepted, we are asking the program committee to put extra effort into identifying those papers that have little chance of acceptance before they go out to reviewers. As happened last year, authors of those papers will receive early notification (likely in October) that their paper has been rejected, along with a short review from the program committee to provide some useful feedback to authors.
New PCS (or “PCS 2.0”)
The team running precisionconference (PCS), the conference management system that CHI and many other conferences have used for several years, have created a newer version of PCS. CSCW and a few other SIGCHI conferences have used it in the past year, and now we are ready to migrate CHI to it. Based on our experiences, the experience for authors, reviewers and committee members should be more seamless and improved over the earlier version. As this is new for most of us, please do bring any concerns or issues to our attention.
ACM and SIGCHI are in transition between the previous Word template that has been used in years past, and new, more accessible, Word and LaTeX templates. As such, for the paper submission deadline of September 21, we are accepting papers in any of these three formats. However, those papers using the previous format that are accepted for publication at CHI, will need to be converted to the new format for the camera ready.
We understand that the same paper may have different page lengths in the different formats – or put another way, a 10-page paper in each of the formats may contain different amounts of content.. Thus, we leave it to authors to choose from the three formats for their paper submission, while still holding to the maximum of 10 pages (excluding references). SCs and ACs are being asked to be forgiving of small issues or errors in the use of the templates.
For those papers that are accepted for publication, the camera-ready version will need to use the latest template. For those who wrote their submission in the original template, the submission will need to convert to the latest template. For those who wrote their submission in the new template but did not use the macros (as we indicated authors should not do), there will likely be some work to convert fully to the latest template.
Our intent is not to force anyone to cut content in order to fit into 10 pages (plus references) when converting to the latest template. How that will actually manifest itself in practice remains to be seen. For now, please concentrate on making your submission the best it can be to improve the likelihood of acceptance, and try not to worry about the potential loss of content in the conversion to the camera-ready version. As such, for the review process, we are instructing subcommittee chairs, associate chairs and reviewers to focus on paper content, and not consider what might be lost in a conversion from an older template to the latest template. This will be made clear throughout the review process, including the creation of initial reviews, the meta-review, online paper discussion, and at the PC meeting.
Update September 13: In converting text from older templates to the latest template for the camera ready, authors will NOT have to cut content to fit within a certain page limit. There should be no loss of content when switching templates, except to address reviewer concerns of course. As before, we will be instructing subcommittee chairs, associate chairs and reviewers to focus on the paper that they were given to review and administer, and not to worry about what length the paper might take in a newer format.
Hopefully this provides clarity to the entire process from submission to reviewing to submitting the camera ready. With about a week before the paper submission date, please focus your attention on making your submission the best it can be. Good luck!
Your papers chairs for CHI 2019,
Anind and Shen