Champion Sponsors

 

 

 

 

 

Explorations of Remote Attendance at CHI

Carman Neustaedter and Anthony Tang
CHI 2019 Telepresence Co-Chairs

CHI has created the opportunity for people to remotely attend the ACM CHI conference via telepresence technologies since 2016. The goal has always been to increase access to the conference for remote participants who would otherwise be unable to attend due to mobility impairments, chronic health issues, temporary travel limitations, or cost issues. This year:

  • The primary way to remotely attend CHI is through the live streaming of talks;
  • In addition, to support social interactions, we have created a programme that pairs up local and remote attendees through a mobile video conferencing setup.

Remote attendance has been carefully explored at CHI and other conferences. It has been carefully thought through and studied over a number of conferences. Ubicomp 2014 in Seattle, USA had 7 people attend remotely using Beam telepresence robots [1]; CSCW 2016 in San Francisco, USA had 19 people attend remotely using Beams [2]; and CHI 2016 in San Jose, USA saw 33 people remotely attend via Beams [3]. Each of these experiences was studied and, ultimately, it was found that remote attendees highly valued being able to use a telepresence robot to remotely be there, move around, watch talks, and engage socially with others. Beams worked especially well for small-scale social interactions with others, like during breaks and in-between sessions. Of course, the experience wasn’t without its challenges. Interactions were not always easy and sometimes local attendees were less excited about the Beams (e.g., blocking one’s view during a talk, being disruptive). But, for the most part, local attendees saw value in helping conferences create a more accessible and inclusive environment for people. Given the overall success, remote attendance continued at CHI 2017 and CHI 2018. Live streaming of talks was added at CHI 2018 in Montreal, Canada and this helped get around challenges with remote attendees finding it hard to see speakers and slides during presentations.

Telepresence robots present pragmatic challenges and other solutions are also needed. Over the years we also have faced pragmatic challenges with using telepresence robots at conferences. Telepresence robots require high Internet bandwidth over WiFi in order to work well. Beams (or equivalent telepresence robots) aren’t available everywhere and so we have had to ship them to each conference’s location from California, USA. This was not always cheap and did not present a globally responsive solution to reducing carbon footprints. While remote attendees helped promote sustainability by not traveling, as ironic as it is, the Beams still had to travel. CHI 2019 proved to be especially challenging when it came to telepresence robots for a number of reasons. Shipping costs to get Beams to Glasgow was high; the conference venue has a variety of levels, some narrow corridors, and different buildings making it hard for telepresence robots to move about; and, WiFi at the conference venue is not available at the bandwidth levels needed by Beams. (But don’t worry – WiFi should be fine for regular attendees! You likely don’t need the > 20 MBps upload and download speeds that a telepresence robot needs.) For these reasons, we decided to explore alternative ideas for remote attendance at CHI 2019.

Over the years, we have brainstormed a large number of different ways to support remote attendance at conferences, ranging from dedicated video conferencing tables in break areas and hallways to situated video conferencing links at the front of presentation rooms to programs that might pair up local and remote attendees. And, that is where we ended up going for CHI 2019.

Firstly, most paper sessions will be live-streamed, free of charge to people taking advantage of this. Secondly, we want to provide access to social interaction with our buddy/human proxy pairing of local and remote attendees.

Research has shown that ‘Human proxies’ can create an enjoyable experience for pairs of people. The idea started back in 2014 when two of Neustaedter’s grad students saw an episode of the US TV show, Arrested Development, where a man on house arrest uses a ‘surrogate’ to be present at work for him. The surrogate wore a head-mounted camera and could do things on the behalf of his boss. While meant to be humorous in the TV show, the students thought it would be fun to try the idea for real in one of their grad courses when one of the students was traveling and couldn’t be in class in person. The experience raised interested technical and social questions, so Neustaedter’s group decided to formally study the use of human proxies in more detail in two university classes. In the classes, pairs of friends tried out the human proxy experience; one stayed at home and attended class through a video conferencing link ‘worn’ by the other friend. The work was published at CHI 2016 and received an Honourable Mention [4]. One of the main conclusions from the study was that students really valued the experience of remote class attendance when paired with a friend. Classroom human proxies were not generally seen as a way to ‘control another person’; instead, it was seen as a way to do pairwise learning with a friend. Students liked being able to interact with their friend while engaging in the class and its activities. Of course, this type of experience still raises many interesting socio-technical questions and, done without the right intentions, could lead to challenging problems around ethics, privacy, autonomy, and much more. That is to say, we are fully aware of the complexities and issues with the idea of human proxies.

Creating networking opportunities through remote CHI buddies and proxies. When the challenges around remote attendance at CHI 2019 emerged, we thought that human proxies may be a possible solution to explore. We envisioned it as a networking opportunity where those local to the conference location could volunteer to ‘bring in’ a remote attendee using a wearable video conferencing system – e.g., a tablet that could be hung around one’s neck, carried in one’s hands, or held at one’s side. The local volunteer could take the remote attendee around the conference venue so they could interact with people and experience the social aspects of CHI like talking with others, seeing demos, etc. To see paper presentations, the remote attendee could watch the live streams of talks on the web. The benefit to the local volunteer would be that they could get to know more people at CHI by attending ‘with’ the remote person, and vice versa. Thus, we saw it as a valuable opportunity for both the local and remote person. We thought being a local volunteer could be especially valuable for newcomers to CHI such as new grad students who may not know a lot of people and want to network with others. It could also be a great way for a lab group attending the conference to involve one of their colleagues who can’t be there. Anybody who is interested can contact us and ask to participate as a proxy. There’s no intention of coercing anybody into being a proxy and proxies can certainly choose what they do to help out the remote attendee.

Given the number of people who have attended CHI remotely over the past two years, we anticipate that there might be up to about 5 human proxy pairs at any one time. Thus, it was not seen as something that would be implemented on a massive scale. Remote attendees will have to pay a small fee to help cover the costs of enhanced WiFi as well as the technical setup. It’s not a means to pay a proxy.

Overall, we see the use of ‘CHI local buddies’ or ‘human proxies’ as a positive bonus for both local and remote attendees to enable social contact and interaction. Our initial CHI web page describing the telepresence experience was unfortunately too ‘functional’ and did not clearly articulate our intentions or what we see as the benefits. The recent discussions have also caused us to rethink some aspects of the implementation by drawing in more volunteers and exploring different mounting/carrying solutions.

We are hopeful that our updates and this blog do a much better job of describing the intended experience and look forward to the community’s support in enabling increased access via remote participation at CHI.

References

[1] Carman Neustaedter, Gina Venolia, Jason Procyk, and Daniel Hawkins. 2016. To Beam or Not to Beam: A Study of Remote Telepresence Attendance at an Academic Conference. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’16). ACM, New York, NY, USA, 418-431. DOI: https://doi.org/10.1145/2818048.2819922

[2] Carman Neustaedter, Samarth Singhal, Rui Pan, Yasamin Heshmat, Azadeh Forghani, and John Tang. 2018. From Being There to Watching: Shared and Dedicated Telepresence Robot Usage at Academic Conferences. ACM Trans. Comput.-Hum. Interact. 25, 6, Article 33 (December 2018), 39 pages. DOI: https://doi.org/10.1145/3243213

[3] Irene Rae and Carman Neustaedter. 2017. Robotic Telepresence at Scale. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 313-324. DOI: https://doi.org/10.1145/3025453.3025855

[4] Clarissa Ishak, Carman Neustaedter, Dan Hawkins, Jason Procyk, and Michael Massimi. 2016. Human Proxies for Remote University Classroom Attendance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 931-943. DOI: https://doi.org/10.1145/2858036.2858184

Newcomers Reception… and the top ten things you should know about going to CHI!

The countdown to May begins and we are excited to meet, greet and have an evening of fun intermingling with all newcomers to this year’s CHI! If you are new to CHI and are not quite sure how to navigate and tame the complex beast that is CHI, come and join us!

We are organizing a fabulous welcome reception for Sunday May 5th at the Conference Centre. Come to mingle with other newcomers as well as people who have been coming to CHI for years as part of our theme of ‘Weaving the Threads of CHI’. This will be a great opportunity to compare notes, get useful tips, make new friends and help set the foundation for a great week. If you’re feeling a bit overwhelmed when you look at all the things going on, this is the meeting for you.

We’ll start the evening off with some welcomes from some long-time CHI attendees. Find out what over 100 CHI attendees think are most important for newcomers to CHI to know!

From 5pm-6pm, CHI veterans, CHI2019 organizers, and Scottish and HCI special interest groups will welcome you and share tips and providing insights and advice. From 6pm, we’ll be joined by the Associate Chairs and Subcommittee Chairs for a joint reception.

Remember to sign up for the reception on your registration form!

While prepping for the event, we started polling ideas from members of CHI Meta, to pick their smart brains and get insights on what they think it’s most important for newcomers to CHI to know. We asked them what do you wish someone had told you? and asked them to vote their favorite options out of a pre-populated list or to add extra options for us to reflect on. Here are the top 10 tips that came out of the exercise:

  1. Don’t try to do everything on the schedule. Take breaks to rest or exercise or whatever you need to do to recharge and be present.
  2. If you find someone’s talk interesting, go up to them afterwards and introduce yourself.
  3. Make new friends. Introduce your new friends to your other new friends.
  4. Go to workshops — great way to network and get inspired (also, it’s a more relaxed setting than a paper presentation)
  5. Don’t feel like you have to fill every slot on your schedule. Make a new friend and go and explore the city!
  6. Go to the panels! You can always read papers later.
  7. As a student, consider signing up to be a Student Volunteer {side note: while too late to sign up for 2019, CHI2020 will offer plenty opportunities!].
  8. If you want to meet someone in particular, reach out in advance and invite them to chat about research at a coffee break.
  9. If you don’t know many people, poster sessions are a great way to meet presenters and other people looking at the same things you are.
  10. Tweet about presentations that inspire you, and read tweets of other CHI-goers. I’ve met several people this way!

There’s much more, and we’ll be presenting this and more information about how to navigate the glorious and exciting event which is CHI. Ultimately… Don’t take it too seriously, engage with people who are interesting (rather than focusing on rock stars) and, if someone blows you off when you try to talk to them, assume it is a problem with them not you — they may feel maxxed out, stressed out, exhausted, introverted, sick, etc.
Be generous and kind, and we look forward to seeing you at CHI!

Daria Loi
Jofish Kaye
Co-Chairs, CHI’19 Newcomers Reception

Countdown to CHI 2019!

The new year is here, registration will soon be open and the countdown to May for a great CHI in Glasgow begins! The content of the scientific programme is shaping up to be very exciting, with the acceptance of papers, case studies, courses, workshops and symposia, doctoral colloquia, etc. already done and the final content getting decided following the last set of deadlines on 7 Jan.

This emerging programme has only happened because of all the fantastic contributions from the CHI community, both as authors/contributors and as reviewers and committee members. Submissions to all the main tracks has grown significantly this year. We want to acknowledge the enormous amount of volunteer work from our track chairs and their teams  – thank you! We also recognise that this has been much more work than usual, for both the chairs/volunteers and authors, as we deal with the impact of numerous changes beyond our direct control, like PCS 2.0, new (emerging!) templates and so on. All of these changes are for good reasons but their implementation has proved more difficult than anticipated. We know this has added work and frustration all around so thank you for sticking with it. We also invite you to volunteer to help improve processes for next time (see the new year email from Helena Mentis, our SIGCHI President).

As we prepare for CHI2019, we’d like to share some of the new initiatives we’re undertaking, many in response to survey feedback from past CHIs. These include:

Weaving the threads: Our theme is ‘weaving the threads of CHI’ and we are trying to encourage this weaving of people in multiple ways to create more opportunities for people to meet and connect:

  • A Newcomers Reception will help to orient new people to the CHI conference experience and to help them meet others in the CHI community
  • Lunch@CHI’ will enable you to have lunch with people who share similar interests
  • CHI Stories will allow you to get to know colleagues in a different way and encourage us all to share our stories
  • We will provide various spaces throughout the venue for people to meet up informally, including some on-site catering for buying lunch and eating together – more to come as we get the space and plans settled.

Equity: A key part of weaving the threads is ensuring that all ‘threads’ are equally welcome and included. Towards this, we now have Equity chairs who work alongside Accessibility, Language Inclusion, Family/Childcare, and the Diversity and Inclusion Lunch chairs to ensure we have a more coordinated and joined up approach to promoting equity. In particular, we hope to provide a network of allies to help increase awareness of the ACM Policy on Discrimination and Harassment and to better support people at the conference.

Industry: We’d love to encourage more industry involvement in CHI and towards this we have great enthusiastic Industry Liaison chairs who are looking at how to better connect with practitioners and the UX community. Keep your eyes open for a Tuesday evening UX event.

Sustainability: We are taking some great steps towards making CHI more sustainable, recognising of course that international travel is still a huge part of getting to conferences. We have some wonderfully enthusiastic Sustainability chairs who are exploring how we can make CHI more sustainable. Some of the actions we have taken include:

  • The all virtual PC meeting saved significant amounts of international travel and produced an excellent programme, and in doing so also enabled more diverse participation of ACs/SCs who otherwise wouldn’t be able to attend
  • We will not hand out a conference bag by default this year. The feedback we get is that these often tend to get thrown away before people even fly home. Instead, we will provide the option of purchasing a high quality reusable bag for those who still want one
  • We are providing sustainable options for merchandise and using more locally-sourced food and drink for the breaks at the SEC where feasible
  • We are reducing the amount of paper we generate by simplifying the print programme to give key overview information. We will use the web site and mobile app for more detailed information, as well as a pdf that people can download in advance. For accessibility or for those people who might need a printed version, you can request it as part of registration.

Evening events: We have created a programme with something on offer all participants every evening: the Opening reception on Monday; the Job Fair, Industry event and CHI Stories on Tuesday;  and a reception at the Glasgow Science Centre on Wednesday evening thanks to the City of Glasgow.

The parties that different organisations host around the CHI conference are outside of our remit and control. However, we do hear feedback about these parties setting up a competitive and exclusive culture. While we have no control over these, we have put in place some Party Liaison chairs whose remit is to talk to party hosts and see what can be done to make events more inclusive.

Remote participation: We will live stream all, or almost all, papers sessions so that those who can’t attend in person can join in. We are trying something new for telepresence, with human proxies for social events, breaks and lunches.

We’ll be asking the chairs of many of the tracks mentioned above to write a blog post to provide more details.

We’re very much looking forward to seeing everyone in Glasgow in May!

Geraldine and Steve, CHI 2019 General Chairs

Explaining the Open Reviewing Process for alt.CHI

Alt.chi is a juried track, with an open review period until 21st January 2019. Anyone can register to review alt.chi papers, and we encourage you to explore the submissions and contribute to discussions. Reviews and discussions will be used in final jury decisions. Notifications will be sent on 30th January, and Camera Ready is 8th February for accepted papers.

PCS, the system used by CHI, doesn’t make the open review process easy, but we have come up with some step-by-step instructions on how to review on the alt.chi page here:

  1. Although this is an open review process, there are a few steps necessary for New PCS to allow access to the submissions:
    Please volunteer to review if you haven’t already. To do this, visit the volunteer page and select SIGCHI CHI2019, and enter a number of reviews for alt.chi.
  2. You will then have access to the “bidding” page. You will see a list of all available titles and abstracts. For any you wish to review, please select “want” or “willing”. You can bid on more papers at any time during the review period.
  3. The chairs will assign you every paper you bid on, unless particular papers are oversubscribed. This is not automatic but is done regularly. You will receive an email confirmation when a paper is assigned.
  4. Once assigned, you will have access to the submission, and any existing discussion and reviews through your reviewing page.

The deadline for reviews is 21st January 2019. However, as reviews are open, we encourage you to submit as soon as possible, to allow for discussion among the other community members. You may revise your review until 21st January.

There are a lot of submissions, and a short amount of time to collect reviews, so please do share the link to the review instructions (above) among your networks. Of course, we hope that you also spend some time reviewing other submissions.

There looks to be a lot of exciting work this year and we can’t wait to see what everyone thinks!

Conor, Ann & Ben

CHI 2019 SV Selection Process

Dear community members,

We recently selected Student Volunteers for CHI 2019. As of now we have 175 lucky students that will get to be SVs at CHI and help us make sure that the conference runs well and smoothly. This represents an acceptance rate of only about 15% as we had 1,204 students enroll to become SVs. While the SV program generally relies on official CHI media posts and word-of-mouth to get students enrolled in the selection process, this year as a way to try to improve the diversity in our lottery pool we have also reached out to all local SIGCHI chapters asking them to share with their students that enrollment was open. The final selection of students is partially random and partially otherwise selected. Let’s break down what that means and the numbers. There are three ways for a student to become a student volunteer at CHI:

PC Meeting SVs

These are SVs that are local to the city where the Program Committee (PC) meeting will happen. These students are locally recruited and selected based on their availability in relation to the required hours for the PC meeting. This year 19 students were PC meeting SVs (~11%).

Nominated SVs

Students can be nominated by members of the Program or Organizing Committees to become SVs. This year 40 slots (~23%) were reserved for these students. We received 227 nominations, but unfortunately 29 of those students did not register at chisv.org, so they were not considered. This means that there were 198 students for 40 slots, an acceptance rate of about 25%. The SVs selected from the nominated students were selected in one of two ways:

  1. 15 of the 40 slots were selected based on the information provided by the nominator. We looked for strong recommendations on the person’s ability to perform SV-related tasks, we looked for opportunities to increase the diversity of the SV group, and we looked for people who would benefit the most from being an SV for the current year.
  2. 25 of the 40 slots were selected through a random lottery.

General Selection SVs

Students can also be selected from the general selection process to become SVs. This year there were 106 slots available and, at the time of the selection, 1,141 students enrolled. This means only a 9% chance of getting in! There are two ways for a student to be selected at this step:

  1. 35 of 106 slots (or 20% of total slots) were selected as “institutional knowledge SVs.” These are students that have been SVs at CHI before, are experienced with a variety of tasks, and can help train the incoming class of SVs. All of them were exceptional SVs in previous years (e.g., always on time AND very proactive AND helpful to others on/off duty AND went above the requirements for their current task).  Many of them are trained in specialized tasks. Unfortunately due to the high competition for a SV slot, we couldn’t accept all students that fit this description. There were many more past SVs we would have loved to have back, but it was important to us to give people who are new to CHI a chance as well. We could only bring back approximately 50% of past SVs that were in this group.
  2. The remaining 81 of the 106 slots (or ~46% of the total slots) are randomly selected from a weighted lottery.

Weighted Lottery

The current lottery system, built into chisv.org, assigns one ticket per student enrolled. We can also increase the number of tickets assigned to a student (a.k.a. influence the likelihood that a type of student has to win the lottery) based on:

  • Whether a student is local;
  • Whether a student has attended CHI before;
  • Whether a student needs a VISA for the conference;
  • The type of program the student is currently in (BA/MA/PhD); and,
  • Whether a student has been a SV at CHI before.

This year, to ensure we gave opportunities to many new students, we granted two extra tickets to people who have NOT attended CHI before and one extra ticket to students who have NOT been SVs at CHI before. Thus, the likelihood to get accepted increases for people with higher ticket counts (up to 4 this year), but students with low ticket counts still have a chance to win the lottery. As all of the factors mentioned above are optional, we did not give extra tickets depending on whether the student is local, whether they need a visa, or their level of study, as we did not believe these aspects aligned with our goals.

Lastly, we are working with a student to upgrade the chisv.org system. This means improving parts related to logistics during the conference, making the interface more inclusive and accessible, and improving the lottery process. We are investigating opportunities for future chairs to tailor the population of SVs, especially to increase the diversity of the SV program. If you have suggestions on aspects that should be built into the new lottery system, please consider taking a few minutes and writing up your opinions here.

And who got in?

While these numbers are not final as sometimes students give up their SV slot (e.g. because of visas, funding, personal reasons, etc.), they represent the SVs that were selected from the process above. The CHI 2019 SVs are:

  • from 39 different nationalities and 100 different universities;
  • almost half first time CHI attendees (85, 49%);
  • largely new to the CHI SV program (126, ~72% have never been CHI SVs);

However, there are two big things to note. All information is self-reported and we rely on students filling them correctly. Also, our incoming SVs are only as diverse as the pool from which the lottery is drawn! We worked hard this year to reach out to all local SIGCHI chapters, but the best way to ensure more diversity in the SV program is to encourage students in your country, university, and department to enroll.

Who are Student Volunteers (SVs)?

Student volunteers have become an essential part in the organization of CHI.  They play a major role in executing structural tasks – especially during the conference. Among other things: we stuff your conference bags; hand out and check badges; show you where to find a paper session, restaurant, bathroom, your lost water bottle, etc; help set up nets for drones, build sculptures out of coke bottles and other exciting demos;  also help figure out where is that missing paper presenter and why, oh why, isn’t the microphone working anymore? Along with many others, the student volunteers put A LOT of effort into helping CHI run smoothly.

SVs are also HCI researchers. Quite a few SVs already publish their research at CHI and have been attending conferences for a while. For others, CHI is a whole new experience, allowing them to see how research results are distributed and how the community interacts. In both cases, being an SV is an incredible opportunity to network with possible mentors, collaborators, and peers.

And here is the tricky bit: SVs are students! We are not trained event managers or AV technicians.  We are volunteers and conference attendees. All SVs agree to an informal contract: in exchange for 20 hours of their time (many put in much more), the conference waives their registration fees and provides daily lunch and breakfast (a lot of tasks start as early as 6:30am, some go well through lunch, and others end as late as 8pm). SVs still have to pay for their own housing and transportation. Workshops and courses are not covered either.

In the scope of their 20 hours, SVs engage in plenty of tasks some of which are simple, while others require certain expertise or training. For this reason we need some people who have done the job before and can teach the job to the next generation of SVs. This is the basic concept: returning SVs show incoming SVs how tasks work, new students come up with new ideas on how to improve them.

And who are the SV chairs?

SV chairs are two senior SVs who have seen the process through multiple years of serving as SVs. The SV chair position is a two year commitment (one junior chair and one senior chair). This is because the CHI SV program is a beast. With 175 SVs each year, coordination with many conference chairs, multiple stakeholders with different needs, serving as SV chair for CHI requires SV experience and training in the role. The first year the junior SV chair observes, learns, and helps with organizational tasks. They learn how to operate chisv.org, how to address different types of  requests, and how to manage an operation the size of a small startup.

Future SV chairs are selected based on their experience as SVs, their graduation timeline, and the specific needs of the conference that year.  The current SV chairs consider several candidates and make a recommendation to the General Chairs of someone they are confident will do a good job in organizing the SV program for the coming years. Sometimes, when there are special needs the General Chairs of future CHIs are brought in earlier in this decision process, to ensure that the “rollover” SV Chair will be able to attend to those needs (e.g. locations where English is not as widely spoken or where cultural norms are significantly different).

As SV chairs, we invest more than 100 pre-conference hours planning SV tasks and ensuring the conference is adequately supported by the SV program. During the conference we spend the majority the conference managing and addressing incoming requests. If you would like to be a future SV chair, make sure you are a SV at CHI (and other conferences as well), have higher responsibilities tasks (volunteer for them!), and that you let us know that you are interested. Every year we go through the process of picking a new junior chair and that person can be you!

What’s been provisionally accepted?

In this blog we consider the big picture from the CHI 2019 Papers track. We look at what’s been accepted, what hasn’t. We also consider how things went with reviewing and look at some author-level data. We will analyze the effect of rebuttals in this cycle in another blog post.

Overall

This year, 2958 papers were left in the system for review. Of these, 705 have been provisionally accepted for the Papers track at CHI 2019. That is a 23.8% acceptance rate, down from 25.7% last year. Eighty-three of these accepts are being shepherded*. The overall mean score for all submissions, including desk and quick rejects, was 2.52 (SD=0.35).

As in previous years, a significant proportion of papers score between 2.0 and 3.0 (inclusive) – this year more than half (1,494 submissions, 50.5%).  Figure 1 shows the distribution of accepts by score. The figure collapses accepted and shepherded papers into ‘A’s and all other decisions (including withdrawal) into ‘R’s. Accepted papers had a mean score of 3.66 (Range 2.38-4.88). This was lower than last year’s mean of 3.73. Forty-two papers with mean scores lower than 3.0 have been accepted (which is precisely double last year’s 21).

The highest-scoring reject had an average of 3.62. There were 117 submissions rejected with a score of ≥3.0 (but only 65 >3.0). There were 224 submissions that scored >2.8 and <3.2. Of these, 70 were accepted. Submissions averaging >3.25 (n=587) were extremely likely to be accepted (97.1%).

Histogram showing the distribution of scores, grouped by decision to accept or reject.
Figure 1. Distribution of scores, grouped by decision.

Reviews

A total of 2799 external reviewers completed 5433 external reviews (Mdn=1, M=1.94). As you will see in the next section, this means that authors outnumber reviewers just under 3:1. The General Chairs, TPCs and Papers Chairs thank reviewers for this absolutely essential contribution. No reviews means no accepts, no accepts means no conference.

The most reviews completed by an external reviewer was 12. Five or more reviews were completed by 189 reviewers. The distribution of completed reviews by a given reviewer is illustrated in Figure 2.

Histogram showing number of reviews completed by external reviewers.
Figure 2. Distribution of external reviewer effort. Most external reviewers complete one review.

External reviewers are only half the picture, though. The other half is the extremely dedicated programme committee, whose associate chairs write reviews and meta reviews and who made the decisions over the 7th and 8th of December. The 400 members of the PC produced 6074 reviews (or around 15 each). Just producing the reviews is a gargantuan amount of work. Unfortunately we don’t have a chart that shows the work that goes into finding external reviewers, managing those reviewers, sending reminders, facilitating discussions, attending the PC meeting, writing final post-PC comments, shepherding, checking camera-ready submissions,… it goes on and on. The General Chairs, TPCs and Papers Chairs are extremely grateful to the ACs for their efforts in putting together CHI’s biggest ever programme.

Histogram showing the number of submissions handled by ACs.
Figure 3. Distribution of AC reviewing effort. ACs generally completed 15 or 16 reviews, split between standard reviews and meta reviews.

Authors

A total of 8,319 authors were listed on submissions, give or take a few duplicate accounts. The vast majority of authors (6,425, 77%) were involved in one submission. Of these authors, 1,462 (22.8%) had their names on an accepted submission. Being on a large number of submissions is uncommon. Only 265 of those 8,319 authors (3%) made five or more submissions. Only 13 authors (0.15%) have their names attached to five or more accepted papers. We congratulate these authors on their success, but note that the vast majority of authors submit one paper and that submission is not successful. Do not be disheartened if things have not gone as well as you might have hoped. You might consider submitting a revised version of the paper to a different SIGCHI conference, many of which will be issuing calls of submissions soon.

Are people who make more submissions generally more successful with them? Only thirteen authors had three or more accepts without a reject. Two authors had four. Conversely, 45 authors made five or more submissions without any success. Figure 4 helps to make this a little bit clearer. It plots, for each author, how many accepts they had, versus the number of rejects they had. Note, authors with more than four accepts (n=13) are excluded from this plot as the published proceedings may allow for their identity to be determined. Authors with more than 7 rejects (n=25) are also excluded.

A plot showing the number of authors who had N papers accepted and rejected.
Figure 4. Individual accept/reject numbers for the CHI Papers track. Authors with more than seven rejects (n=25) or more than four accepts (n=13) are not shown. Note the density of points for each combination (e.g., 2 accepts, 1 reject) reflects its frequency across the whole set of authors. The more square the ‘patch’ appears, the more authors there are with a given combination of accepted and rejected submissions.

*ACs can require changes to papers before the camera-ready submissions are made. A small number of papers are assigned a ‘shepherd’. This is an AC who helps the authors make changes to the manuscript in a way that meets the requirements of the PC and reviewers. It often happens when a paper needs specific changes that are carefully made in order for it to be it accepted.

CHI 2019 Reviews and Rebuttals

This blog post is about the current scores of submissions to the Papers track immediately after the completion of reviews.

You will have recently received your first indications of how your submissions might fare in December’s programme committee (PC) meeting. Most papers will have received four reviews. Two reviews have been provided by external reviewers. One of the PC members, the paper’s ‘2AC’ has also provided a review of your submission. The final review is a meta review provided by the paper’s ‘1AC’, another member of the committee.

Across the 2960 papers, 2256 papers (76.2%) received precisely one 1AC review, one 2AC review and two external reviews. Other submissions had a variety of review configurations. For example, 164 submissions received a single 1AC review, as part of the quick/desk reject process. Another 326 papers received three AC reviews, in four cases because the paper only had one external review, but in the other cases because the ACs might have wanted an additional opinion on a paper that split opinion, or a technical aspect of a submission. A total of 409 papers received five or more reviews. Nine papers received six reviews.

A total of 2960 papers were submitted. This year the committee has made more use of the quick/desk reject option available to them. A total of 283 papers (9.6%) have already been rejected. This is a substantial increase from 55 last year. They were rejected because:

  • They broke this year’s length restrictions (24)
  • They broke the CHI’s anonymity policy (71)
  • They were considered out of scope for the conference (8)
  • They did not meet the conference’s (very relaxed this year) formatting requirements (1)
  • In the opinion of the committee they did not meet the quality threshold to go out for review (179)

The lowest score a reviewer can assign a submission is one, and five is the highest. Reviewers also self-report their expertise as it relates to a given submission. This is rated on a scale from one (least knowledgeable) to four (most knowledgeable). Together, these scores give you an idea of how your paper has fared.

Each submission has an ‘Overall Score’, which is the mean of the scores given by the ACs and reviewers of a submission. The mean of these overall scores is currently 2.48 (SD=0.76). If your paper has an overall score of 3.5 or better, it is currently in the top 10% of submissions by score. Only 105 papers scored a mean of 4.0 or better (3.5%). The distribution of scores is illustrated in Figure 1. Overall, scores are lower on average this year, with this year’s mean of 2.48 lower than the mean score of 2.56 at the same stage last year. In 2016 the mean score at the same point was 2.63.

In other words, overall reviewers lean toward rejecting most papers, with around a quarter of papers receiving a neutral or positive response from reviewers. Experienced submitters to CHI will recognise the distribution of scores. It is quite similar each year. If this is your first time submitting to CHI, then the score distribution should help you contextualize the ratings your submissions have received. The main takeaway is that almost half (1473, 49.8%) of all submissions score ≥2.0 and <3.0.

Chart showing distribution of review scores for the initial round of CHI 2019 reviews.
Figure 1. Histogram of mean scores for submissions to the Papers track. Marked is the current mean score, 2.48; the score needed to be in the top 25.7% of submissions (i.e., last year’s acceptance rate), 3.0; and the 90th percentile (i.e., top 10%), 3.5.

A huge amount of effort goes into reviews. The latest data we have shows there are 11,175 reviews in the system. These reviews comprise 6,137,208 words. The longest was a whopping 4550 words. The shortest was 34 words. There were 969 reviews of over 1000 words, and the mean length was 549 words (SD=332). The distribution of review length is illustrated in Figure 2.

Chart showing the distribution of CHI 2019 review length, in words.
Figure 2. Histogram of mean review lengths for submissions to the Papers track. The longest review was 4550 words, the shortest 34. Note the x-axis uses a log scale.

We also had a look at whether there was a relationship between overall score (i.e., mean score) and mean review length. (We use means because an individual only has nine score choices and the plot is a little dull, then.) We plot this relationship in Figure 3. Note that submissions with very high scores and submissions with very low scores tend to receive the shortest reviews, on average.

Plot showing the word length of a review versus the review score.
Figure 3. Mean words in reviews against mean score for all submissions. Note that reviews around the mean submission score receive the longest reviews.

One of the least visible parts of the review process to authors is the discussions that reviewers have about submissions in PCS. Before rebuttals were sent out, submissions had received between zero and 25 discussion comments with a median of two. One might expect that as reviewers diverge more in their scoring of a paper, so the amount of discussion they have increases. Figure 4 suggests that this is indeed the case.

Plot showing the number of discussion comments a paper receives, plotted against its score.
Figure 4. Standard deviation of reviewer scores plotted against the number of discussion comments made on a given submission. The greater the variation in scores, the great the amount of discussion a submission generates.

You have a short window (see the submission page for specifics about timing) to submit your rebuttal. There is no requirement that you submit a rebuttal, although reviewers and ACs are often pleased to receive some kind of acknowledgement of the time and effort they have spent on your work.

People often wonder whether it is ‘worth’ submitting a rebuttal at or below a certain score. Last year, the overall mean for scores did not move after rebuttals, but this does not mean that individual scores do not move. They can and they do. If the ACs for one of your submissions have encouraged you to write a rebuttal then it might make sense to do so. If nothing else, a rebuttal gives you the chance to consider the perceived limitations of your submission so that you can fix them, whether for CHI 2019 or another venue.

A number of members of our community have provided views on how to write a rebuttal. Here are a few that were suggested in last year’s blog:

This list isn’t exhaustive. There is other great advice floating around, often tagged with the CHI 2019 hashtag.

CHI 2019 Please Review!

The CHI 2019 Papers track, which is bigger by submission volume and organizational effort than the rest of the CHI tracks combined, closed last Friday. In this blog post we look at what has happened so far with submissions and to give you an idea of what will happen next and to encourage you to be part of the ‘chi circle of kindness’ by contributing to the community review effort.

On the 14th of September, 3784 abstracts were submitted to PCS, the conference’s submission system. By the deadline on the 21st of September, there were 2966 complete submissions left in PCS. This means that 22% of submissions at the abstract phase were either deleted or left incomplete by the deadline.

A map of the world that highlights the 67 countries from which a submission was received.
CHI 2019 received 2966 complete submissions from authors with affiliations in 67 countries.

Excepting any submissions that are rejected early in the process, these 2966 papers (a 14% increase in complete paper submissions compared to CHI2018) will each receive four reviews: two external reviews, one internal review from the 2AC and one meta-review from the 1AC. Therefore, the submissions will require almost 12,000 reviews. This is a huge amount of work.

We need your help as part of the community to get through this work. Last year 2651 reviewers produced 5105 reviews. But most people only wrote one. This meant that 1053 reviewers produced 3621 reviews. Remember that this year we will need additional external reviews to ensure everyone’s submissions get the kind of attention that makes CHI a high-quality venue for publishing.

So we’d love you to volunteer to review here, especially so if you are an author as each paper you have submitted generates four reviews: https://new.precisionconference.com/review_volunteering

We’re doing all of this with the added ‘excitement’ of using PCS 2.0 for the first time for CHI – please be patient as we deal with the inevitable teething problems that come with using a new system. And dealing with teething problems also takes a huge additional effort over and above the huge volunteer effort already involved in making CHI happen. For this we already owe a huge debt of thanks to the papers chairs (and many other members of the committee and the SIGCHI EC) who have been working on our behalf to make it all happen!

TPC chairs for CHI 2019, Anna Cox and Vassilis Kostakos
Analytics chair for CHI 2019, Sandy Gould

Volunteer as an LBW AC

Interested in being an Associate Chair for the CHI 2019 Late Breaking Work track? This is a great way to get involved with the conference and contribute to the CHI community. Volunteer here before September 30th.

Updates

12th February 2019
Read about the Newcomers Reception and top 10 things to know about CHI.

1st February 2019
Check out the list of accepted courses.

31st January 2019
Registration and hotels now available!

25th January 2019
Countdown to CHI 2019!

7th January 2019
Financial support is available for students.

14th December 2018
Check out the list of accepted workshops and symposia.

Shortcuts

For Attendees
Registration
Hotels
Venue

For Exhibitors
Registration

Scroll Up