|
Before Wed, 11 Sep, 20:00 ET
|
- SCs communicate with your co-SCs to decide how you're going to approach AC assignment, and (for Health / Interaction Beyond the Individual / Understanding People / Interaction Technologies / Design) how you're going to approach split assignment.
≫show details for SCs
Two of your first tasks as SCs are assigning papers to splits within your subcommittee (for Design / Interaction Beyond the Individual / Understanding People), then assigning ACs to submissions (all subcommittees). These stages operate on very tight timelines, so it's crucial that you have a plan for how your subcommittee wants to handle these tasks. We'll give you all the details about how to check the right boxes in PCS, but there's no "right" way to split this work up among SCs, so please start communicating now about how you want to approach this, including who is available when. For splitting, is one member going to get all your co-SCs on the phone and all do it in real-time? For AC assignment, are you going to take turns with your co-SC (this happens at the level of individual splits, so you'll only coordinate with one co-SC at this point), or will you be on the phone together?
- SCs: For those attending in person, obtain visa for PC meeting (if required). Here's some guidance for figuring out whether you need a visa.
- SCs book flights to Indiana, USA.
|
|
Friday, 13 Sep, 20:00 ET
|
- Title and abstract submission deadline.
- Paper Chairs identify any subcommittees that need to downsize or upsize.
|
|
Saturday, 14 Sep
|
- Abstracts become visible to SCs.
|
| Friday, 20 Sep, 20:00 ET |
Material submission deadline |
| Saturday, 21 Sep - Sunday, 22 Sep |
- SCs identify any obvious rejects (not actual papers) and flag to papers chairs.
≫show details for SCs
Jokes, ads, submissions without PDF files, material not in English, material not in any of the CHI Proceedings Formats, and anything else that obviously has no chance and should not take additional time from anyone involved with the conference. Raise these to the papers chairs by email; we’ll sign off and delete them from PCS. Being over-length is grounds for rejection at this stage. Fairly liberal bending of the format (e.g. wrong fonts throughout) is annoying, but OK at this stage and does not fall under "obvious reject", if it appears to be based on the CHI Proceedings Format. Not being topically appropriate for CHI (but otherwise being a well-formed submission) is not grounds for rejection at this stage; that will happen at the AC desk-reject stage.
- SCs email ACs to list SC conflict papers.
≫show details for SCs
Once you and your co-SCs have your list of conflicts, you should email this list to your ACs, and remind them to keep it handy: they should be checking it prior to any communication with you about specific papers. ACs cannot access this information through PCS.
- SCs identify papers that were initially assigned to their subcommittee that should be transferred to the secondary subcommittee chosen by the authors. As much as possible, we should honor authors' choice as much as we can, but for papers that's really not suitable, we can allow it to be transferred to a third subcommittee.
- Paper chairs run the plagiarism detection routine to identify any paper with plagiarism problems, and pass the results to SCs to double check as candidates for desk-rejects.
|
| Saturday, 21 Sep - Monday, 23 Sep |
- ACs bid on papers assigned to their subcommittee in PCS.
≫show details for ACs
To bid on submissions:
- Log in to PCS 2.0, click "Reviews" and click "Review (as a committee member)"
- Click "Bidding"
- For all papers you are in conflict with, select "conflict"
- For at least 15 papers, select "want"
- For at least 15 papers, select "willing"
- ACs select at least 15 papers as “want” and 15 papers as “willing”, for a total of at least 30.
- ACs specify their conflicts in PCS.
≫show details for ACs
Declaring conflicts with papers is helpful, but we know you can't necessarily find every paper that you may be conflicted with, so don't spend too much time on this. Please do declare your own submissions as conflicts.
To declare your conflicts:
- Log in to PCS 2.0
- Click "Reviews" at the top of your home page
- Click "Review (as a committee member)"
- Click "Conflicts"
You will see two huge lists: a list of every institution that submitted to your subcommittee, and a list of every author that submitted to your subcommittee. Don't go crazy clicking every imaginable conflict; this is just a way to pre-empt papers you're conflicted with from getting sent to you and slowing down the process when you have to un-assign yourself. Spend a little time checking your home institution and any authors you know who submitted who are obvious conflicts. If it takes you longer to declare institution/author conflicts than it's taking you to read this paragraph, you're working too hard.
The ACM conflicts policy is here.
|
|
Before Monday, 23 Sep, 20:00 ET
|
- SCs handle AC rebalancing issues based on feedback from Papers Chairs.
≫show details for SCs
After papers are in we will let you know if you are over or under on the number of ACs you need. The number of ACs we told you to recruit was based on projections from the last few years, but things may vary. If you have more ACs than needed then you can let someone off the hook. If you need more, we will tell you how many.
- SCs identify any papers that should be in a different subcommittee and flag to papers chairs.
≫show details for SCs
If you find a paper that you feel your subcommittee does not have the right expertise to review, but another subcommittee does, please contact us ( papers@chi2020.acm.org) so we can do the switch. This should be relatively rare; if it's close to 50/50, even if you would have rather seen a paper submitted to a different subcommittee, we should respect the authors' choice.
- SCs for those committees with splits, split papers across their subcommittee splits and email papers chairs about splits’ composition if changed for rebalancing.
- SCs declare conflicts in PCS and pass responsibility to co-SCs.
≫show details for SCs
You will find a ‘Declare chair conflicts’ link on your main page. You only need to declare conflicts with papers in your subcommittee.
Ideally you will be able to pass over the list of conflict papers to your co-chair. This happens by email; there is no formal mechanism for “owning” a paper as an SC; the point is just to make sure that you pass a list of conflicts along to ACs, so if they have questions, they know which SC not to ask. If your co-chair is also conflicted, please talk to us.
Policy for SC conflicts
Because there is one level of indirection between you and reviews or discussion of the papers, we use a slightly loosened definition for conflict of interest for SCs that avoids some of the more tenuous connections of the full policy. For declaring conflicts inside the system we suggest you declare "core conflicts" only:
- Papers you (co-)author
- Papers with an author at the local branch of your institution
- Papers (co-)authored by current close collaborators
- Any paper you feel you either can't judge impartially, or otherwise represents a conflict you feel you really should declare.
So you only have to declare conflicts with local colleagues and your close collaborators. Otherwise you won't be able to chair the meeting as you'll be outside all the time!
|
|
Before Wednesday, 25 Sep, 20:00 ET
|
- SCs assign ACs (1AC and 2AC) to papers.
≫show details for SCs
This is a central and critical job of an SC. The most important thing in the review process is getting good reviewers assigned, and that starts with getting ACs who are knowledgeable about the topic of each paper assigned, so despite the size of the task, please take this assignment process very seriously.
The bidding should help in this task.
Please DO NOT send ACs notifications for these assignments at this point – we need all ACs to start their work together on September 27.
- For the subcommittees with splits, the SCs will have to first provide the Papers Chairs with the AC composition of their subcommittee splits. We will need to update PCS with this information before papers can be assigned to ACs.
|
|
Wednesday, 25 Sep, 23:00 ET
|
|
|
Thursday, 26 Sep - Tuesday, 1 Oct
|
- ACs look for conflicts and other difficulties, optionally swap problem papers, notify SCs.
≫show details for ACs
The first thing you should do when you log in to PCS is click each paper to check for conflicts. If you're conflicted with a paper, please request for a swap to your SCs. If you are not conflicted with a paper, please proceed to assigning reviewers.
- SCs track swaps and ensure all papers have a 1AC and 2AC assigned in the end.
- 1ACs allocate two high-quality external reviewers to each paper (for batch release later). Different from last year, 2AC will no longer select reviewers.
≫show details for ACs
All review requests will be going out in batch on October 3rd. Therefore, during the current reviewer selection phase, you should not be communicating with reviewers. That means please don't email reviewers to say "hypothetically, are you reviewing CHI papers this year?". The only exception: if a paper really requires a reviewer way outside the CHI community (someone's novel input device needs a physicist reviewer, or someone's hospital-based study needs a clinician reviewer), it's OK to contact them to tell them what CHI is, communicate our expectations for reviewers, and ask if they are up for reviewing papers. If you don't need to tell them what CHI is, you shouldn't contact them at this point.
For papers for which you are the 1AC you select two reviewers. To help focus on the importance of this task, we are asking that you be prepared at the PC meeting to give an explanation to the rest of the subcommittee for why you picked the reviewer you picked. The goal is to have exactly two review requests in place for each paper by the release date of Oct 2, so please do not invite more than two reviewer for each of your 1AC assignments. However, you might want to keep your own list of potential backup reviewers for your assignments.
You are looking for reviewers who will make substantive insightful comments, and who have the perspective to evaluate how interesting results are and whether they are sufficiently relevant to (some part of) HCI as a field. Your first line of attack for finding good reviewers may be your own knowledge -- if the paper is in your area (or close to it), you may be able to directly think of a good candidate that you already know to be an expert. Another excellent way to identify potential reviewers is to consider authors of previously published results on the topic. You will likely find some of these publications in the references of the paper itself. Searching for related work in the ACM Digital Library and other search engines is also typically very helpful. Keep in mind that publishing a single paper on a topic, even at a competitive venue such as CHI, might not mean a person is an expert, and that different authors may have contributed different things to a particular paper. For potential reviewers you are not familiar with in advance, it can be helpful to try to have a look at their overall research record through their web presence.
We do not recommend using the search tool within PCS as keyword matching does not generally yield good matches between papers and reviewers. Please don't request reviews from other ACs or SCs. You can find the full committee list on the CHI '20 website. Nothing in PCS will prevent you from doing this, but you will notice that if you assign a review to a committee member, he/she will appear as a 2AC, rather than an external reviewer, in which case you should unassign.
To actually make an assignment:
- Go to your reviewing home
- Click on the name of the paper
- Type a name (or email address) into the "Assign reviewer" field
- If there are multiple matches, click the one you want
- ACs identify papers as desk/quick reject candidates, notify SCs with a short paragraph describing reasons for desk/quick reject. A description of the early reject process and criteria can be found here.
≫show details for ACs
Now you'll start skimming through your papers to select reviewers, starting with your 1AC papers. The first time you open each paper, check quickly for the maximum length violations: 10 pages, not including references. If you find any, notify your SCs and stop working on this submission; it will be rejected. A paper does not have to be 10-pages long. It can be shorter. Be sure that your reviewers have acknowledged this. The contribution size must be judged regarding to the paper length.
Also look for severe anonymization issues. We think we've handled most of these, but if you find any, notify your SCs right away.
And, in your first pass through a paper, look for clear candidates for "desk rejection", i.e. rejecting papers without recruiting external reviewers. We expect that catastrophically bad writing will be the most common reason for desk rejection. It's very hard for us to lay out objective criteria, but subjectively, it's "there is a zero percent chance that this paper will get in, and you would not feel comfortable using a qualified reviewer's time to handle it when there are some many papers to review."
With those identified, communicate and discuss with other AC assigned and come to an agreement. Once agreed, email your SCs to identify these papers, and write up short rationale referring to the desk or quick reject criteria and communicate to SCs (be careful about conflicts). The SCs will look at all the DR/QR candidates, and make a decision themselves. Once all ACs and SCs agree with the DR/QR decisions. The SCs email the paper chairs about their recommendation.
Minor formatting issues, and even (for example) super-egregious reference formatting violations, are not grounds for desk rejection. Papers that aren't vaguely in 2-column ACM format, or papers that clearly cheated the format to get around length restrictions, are candidates for desk rejection.
- SCs review desk/quick reject candidates, and make decisions.
≫show details for SCs
When scanning papers, please keep an eye out for submissions that have major problems (e.g., ½ page in length, broken fonts, gross formatting violations, etc.) or otherwise have no chance of being accepted.
If the 1AC or 2AC identifies a candidate for desk rejection, they should first communicate their desk reject recommendations to their SCs by email. Both the 1AC and 2AC will read the paper. When a paper is raised for desk rejection, it will go to both the SCs and the papers chairs (with appropriate handling of chair conflicts).
|
|
Wednesday, 2 Oct
|
- Batch release of review requests to reviewers. We will release all of the review requests to reviewers at the same time, to give ACs a fair chance of getting who they want as reviewers.
|
| Thursday, 3 Oct - Friday, 4 Oct |
- SCs calibrate desk/quick reject candidates (they can ask more desk/quick rejects if the total number of desk rejects is below 20% for the subcommittee, or ask for reconsideration if they feel some desk/quick rejects are not appropriate). SCs may filter out some of the early rejects if the number is greater than 20% and can disapprove any candidates if they feel it is not appropriate for DR/QR. When in doubt, SCs can email the paper chairs to discuss any early reject candidates
|
| Saturday, 12 Oct |
- Hard deadline for all desk/quick reject reviews (after SC calibration). If the review is not ready or not in good enough quality, the quick reject decision is reverted, and ACs need to find reviewers for the quick-reject paper candidates.
|
| Tuesday, 15 Oct |
- All desk/quick reject notification sent out to authors.
|
| Before Thursday, 31 Oct |
- SCs (and/or their SC assistant) should go over all of the technology with their ACs
|
|
Thursday, 3 Oct - Thursday, 31 Oct
|
- ACs (and SCs) ensure that each paper gets two external high-quality reviews: track progress and find replacements for reviewers with the shortest delays as possible.
≫show details for SCs
SCs need to take a particular concern for monitoring and encouraging consistently high quality across the review process, e.g., checking that the reviewer selections are experts and not just past students or friends, that there are (only) two reviewers, etc. We ask that SCs particularly mentor new/young ACs. We will leave the details of how you do this to SCs, but we can provide help if required.
You might want to ping ACs who haven't gotten all their reviewers lined up in a reasonable time for a status report, as a means to make sure they are working on it. We really need 2 high-quality external reviews plus the 2AC review for every paper going into rebuttal.
To monitor the requests on the Submissions page:
- Sort by the "Reviews total" column to see which submission do not have enough reviews. The "Reviews total" include reviewer and committee member reviews.
- Sort by the "Reviews tentative" column to see which submissions still have tentatively-assigned reviewers (i.e. reviewers who have not yet agreed to review). You can prompt the AC of each such submission to follow up with the tentative reviewers. Or you can send a reminder email to all tentative reviewers from the Send Email page.
- Sort by the "Reviews left" column to see which submissions still have unsubmitted reviews. These include the committee member reviews.
To monitor the requests on the Committee page:
- Sort by the "Reviews Remaining" column to see which committee members have many reviews left to do. Incomplete reviews are also shown as yellow boxes in the "R" (for "Review") columns on the right. You can send email to particular committee members by clicking on their name, then on their email address on the next page.
Early in this period, we recommend that you remind yourself and your ACs of the following guidelines that we use to communicate our standards to the CHI community:
≫show details for ACs
Until all reviews are in, the 1AC is responsible for the both external reviewers he/she assigned. As the review process unfolds, you should track your outstanding reviews for the papers you are in charge, send a personal reminder about a week before the deadline, and read over the ones you have received.
After a few days, check on reviewers who remain as “tentative” in PCS, i.e., reviewers who have not clicked “accept” or “decline” yet. In your list of review assignments, you should see a Revs column which shows 4 numbers (#completed/#assigned+#tentative+#stored). #tentative shows the number of pending review requests.
If it looks like you may have lost a reviewer, either because you get an explicit notification from PCS, an email from the reviewer, or any other indication that the reviewer is declining or is unavailable (e.g., time has passed with no communication at all), you should recruit a replacement review. You can do this directly through PCS, and there is no batch release process for replacement reviewers (your requests will go out immediately). At this stage, it is OK to contact potential reviewers by email.
Monitoring and encouraging quality reviews is the second critical part of your job once reviewers are recruited. The most common complaints we receive from authors are that review decisions are not well justified or that reviewers have not used an appropriate tone or language. If you find reviews that you think are of lower quality than you would be proud to stand up and defend at the PC meeting, you may want to (very gently and diplomatically) suggest to them that they e.g., extend the rationale given for their score, alter the tone or language they use, etc. This is important to ensure that all judgments are well justified, authors can understand them, and they can be weighed against opposing views.
You might want to remind yourself and your reviewers of things like:
|
|
Friday, 1 Nov
Note: There are two types of 3AC — 3AC by score & 3AC by request
|
- Reviews due back from reviewers and 2AC.
- 1ACs (and SCs) track missing reviews & quickly resolve (possibly with replacement reviews).
≫show details for SCs
Track the status of papers and ensure that no ACs have a pattern of problems (missing reviews). Prod ACs for status on replacement reviews if needed. Again, it is critical that we have 2 high-quality external reviews plus the high-quality 2AC review for every paper going into the rebuttal process.
≫show details for ACs
Quickly resolve missing reviews. Track missing reviews & quickly resolve issues with reviewers you that have recruited as 1AC or 2AC. In rare cases it may be necessary to find last-minute replacement reviews. It is critical that we have two high-quality external reviews for every paper going into rebuttal.
- Papers chairs assign preliminary “discuss” status based on scores, indicating which papers need a 3AC assigned. Note that there are two types of 3ACs. The orange type is the 3AC assigned based on score criteria, which is done before the rebuttal. The yellow type is the 3AC assigned after the 1AC’s meta-review, which will enter their review after the rebuttal due to time constraints. They have different deadlines in completing their reviews.
≫show details for SCs
Assign a 3AC to all papers that have a discuss status, being sure to balance the 3AC load across all of your ACs.
≫show details for ACs
The Papers Chairs will determine a preliminary “discuss” status for papers (i.e., papers that may be accepted, and therefore discussed at the PC meeting), based on numeric criteria (we’ll let you know what these criteria are on 3 Nov). Papers with a “discuss” status will get an additional meta-review from a newly assigned 3AC (deadline: Nov. 26). Papers without a “discuss” status will only get a meta-review from the 1AC.
- ACs initiate reviewer discussion as needed.
≫show details for ACs
Use the PCS reviewer discussion system to encourage discussion if the reviewer opinions on a paper are split. This is meant to help you formulate a thorough meta-review that appropriately captures reviewer opinion, and also to make sure that no reviewer missed or misunderstood something that another reviewer is able to clarify. The discussion period closes on 13 Nov.
|
| Saturday, 2 Nov |
- SCs assign 3ACs to papers that meet score criteria
|
| Monday, 4 Nov |
- ACs ensure to facilitate a discussion
- Discussion between reviewers
|
| Wednesday, 6 Nov, 11:00 ET |
- Papers chairs share videos demonstrating the technology for the PC meeting for the SCs and ACs (Zoom, PCS 2.0, PCS Chair tool, slack, Trello)
|
| Friday, 8 Nov |
- 3AC reviews, assigned based on score criteria are due
|
| Tuesday, 12 Nov |
- Hard deadline for all reviews (external, 2AC, 3AC), except 3AC reviews assigned after 1AC meta-review are complete.
|
|
Before Wednesday, 13 Nov
|
- ACs write 1AC meta-reviews for all of their 1AC papers and check “discuss” box for papers that need it
≫show details for ACs
Once reviews for a paper are available, the meta-review is your most important task as an AC. Meta-reviews should contain some discussion of how you personally see the paper. However, the meta-review is primarily a summary description of the external and 2AC reviews, indicating points where the reviews agree and disagree. The score you give for the meta-review should normally reflect the underlying reviews. As a guide, you should likely start with the average of the review scores and then think about what adjustments from that might be justified based on resolving conflicts and weighting of the importance of particular points raised. We are asking you to use your expert judgment here, but you should not be completely substituting your opinion for that of your reviewers. Your meta-review should include your comments on the paper and must include a synthesis of the reviewers’ views and your opinion of those. It should be clear to the authors when they read the meta-review which comments are the opinion of the AC and which comments are a synthesis of the external reviewers. Meta reviews to be completed by 13 Nov.
If a paper is going to be discussed, you should clearly identify in your meta-review the points you feel it is most important for the authors to address in rebuttal.
If a paper is clearly not going to be accepted, do your best to convey this diplomatically, and with the empathy we give to all authors (we’re all authors too). With that said, don’t give “false hope” if a paper is a clear reject. Phrases like “if you continue this work for submission to another venue in the future, reviewers suggest that…” are helpful and encouraging.
Papers for discussion or needs a 3AC
In the Recommendation for PC Meeting field, select the “Discuss” option for papers that need it. If you come across a paper that does not meet the criteria for discussion/3AC assignment (i.e., an “early reject” paper), but you feel it merits further discussion,select the “Discuss” option in your meta-review by 13 Nov.
- SCs assign 3AC to papers that did not meet 3AC criteria, but where 1AC has indicated the paper needs to be discussed
≫show details for SCs
Assign a 3AC to all papers that 1ACs have marked for discussion, being sure to balance the 3AC load across all of your ACs.
- SCs review all meta-reviews for quality
≫show details for SCs
SCs review the meta-reviews, giving feedback about changes they'd like to see before reviews get sent to authors.
|
|
Wednesday, 13 Nov
|
- Deadline for 1AC meta-reviews and recommending papers for discussion (to indicate that a 3AC review is needed for a paper that otherwise didn’t meet the criteria).
- Deadline for SCs to assign 3AC for all papers marked Discuss in the Recommendation for PC Meeting field.
|
|
Friday, 15 Nov
|
- Reviews released to authors for rebuttal.
|
|
Friday, 22 Nov
|
- Author rebuttals due.
≫show details for SCs
For papers with rebuttals, ask the ACs to contact reviewers to strongly encourage them to read the rebuttals, acknowledge the rebuttal in their reviews, and update reviews accordingly. ACs should do the same.
|
|
Tuesday, 26 Nov
|
- Deadline for 3AC reviews (assigned after 1AC meta-reviews are in).
- 1ACs re-engage reviewers, 2AC, 3AC, in discussion of each submission, as needed, and ensure that reviewers update their reviews to acknowledge rebuttals. If rebuttal affects scores, the scores should change during the discussion period, which ends 4 Dec.
≫show details for ACs
For papers with rebuttals, 1ACs should contact reviewers to strongly encourage them to read the rebuttals and discuss on the PCS "discussion board". 1ACs should also ask reviewers to acknowledge the rebuttal in their reviews, and update reviews accordingly. 1ACs and 2ACs should do the same. Since the authors will have seen your meta-review and external reviews, add your comments in a “Post Rebuttal Additional Comments:” section and encourage your external reviewers to do so too.
|
| Tuesday, 3 Dec |
|
|
Before Wednesday, 4 Dec
|
- 1ACs indicate a preliminary accept or reject status for each of their papers (and ideally sooner). For preliminary rejects, please click the 'R' in PCS, for the preliminary accept, please do not set the 'A' yet, but do this during the PC meeting, after AC1 have presented the paper.
|
| Before Friday, 6 Dec |
- ACs update meta-reviews, any changes in review scores, and final “discuss” status due.
- ACs get ready to justify your choice of reviewers, provide a summary of the paper and reviews, and a recommended decision (accept/reject)
|
|
Thursday, 5 Dec (18:00 ET) - Sunday, 8 Dec (~15:30 ET)
Time Slots:
- 06:00-10:00 (Pacific Time)
- 09:00-13:00 (Toronto)
- 15:00-19:00 (Central European Time)
- 22:00-02:00 (Hong Kong)
|
-
PC meeting (Thursday - SCs only; Friday to Sunday - SCs and ACs)
≫show details for ACs
After the meeting you will need to make final updates to the feedback that will be returned to the authors, and mark in PCS when this is ready to be released to the authors. This needs to happen very quickly to meet our schedule, so you probably want to try to have most or all of this done before you leave the meeting. At this stage it is important that you provide information to the authors that enables them to understand why the decision that was made was made. If changes are required for a conditionally accepted paper to become acceptable, you need to provide clear feedback about the expectations/requirements.
|
|
Monday, 9 Dec
|
- Notification of decisions released to authors.
|
| Thursday, Dec 12, 12:00 (Pacific time), |
- SCs make sure that all ACs have finished their meta-reviews including comments from the PC meeting.
- 1ACs check the “Yes, final reviews are ready” checkboxes once done.
- SCs will monitor if all papers have the checkboxes checked.
- Shepherds need to contact the authors about the shepherd process and timeline
|
|
Friday, Dec 13, 12:00 (Pacific time)
|
- Reviews released to authors.
|
| Friday, Dec 27, 12:00 (Pacific time) |
- Shepherded papers need to submit a revised version to their shepherd
|
| Friday, Jan 3, 2020 |
- Final feedback from shepherds to authors before camera-ready deadline
|
| Wednesday, Jan 8, 12:00 (PT) |
- Camera-ready deadline for authors
- AC needs to check the submitted paper and click the 'Final acceptance of camera-ready' box in PCS2.0
|
| Saturday, Jan 11, 12:00 (PT) |
- Deadline for ACs to check the 'Final acceptance of camera-ready' box
|
| Monday, Jan 13, 12:00 (PT) |
- Final acceptance of all papers send to authors by papers chairs
|
