|
June 2021
|
Finish Recruiting the ACs
|
|
July-August 2021
|
Programme Committee Roundtable Discussions (details TBA)
|
|
August 2021
|
Technological Walkthrough (details TBA)
|
|
Before Sept 2021
|
- SCs communicate with your co-SCs to decide how you're
going to approach AC assignment,
and (for Design / Health / Interaction Techniques, Devices, and Modalities /
Understanding People / User Experience and Usability) how you're going to approach
split assignment.
≫show details for SCs
Two of your first tasks as SCs are
assigning papers to splits within your subcommittee (for Design / Health /
Interaction Techniques, Devices, and Modalities / Understanding People / User
Experience and Usability), then assigning ACs to submissions (all
subcommittees).
These stages operate on very tight timelines, so it's crucial that you have a
plan
for how your subcommittee wants to handle these tasks. We'll give you all the
details
about how to check the right boxes in PCS, but there's no "right" way to split
this
work up among SCs, so please start communicating now about how you want to
approach
this, including who is available when. For splitting, is one member going to get
all
your co-SCs on the phone and all do it in real-time? For AC assignment, are you
going
to take turns with your co-SC (this happens at the level of individual splits,
so
you'll only coordinate with one co-SC at this point), or will you be on the
phone
together?
|
|
Thursday, 2 Sep 2021
|
- Title and abstract submission deadline.
- Paper Chairs identify any subcommittees that need to downsize or upsize.
|
|
Saturday, 4 Sep 2021
|
- Abstracts become visible to SCs.
|
|
Thursday, 9 Sep 2021
|
- Material submission deadline
|
|
Friday, 10 Sep 2021 - Saturday, 18 Sep 2021
|
- SCs identify any not actual papers and flag to Papers
chairs.
≫show details for SCs
Jokes, ads, submissions without PDF files, and material not in English.
- SCs and ACs to declare
conflicts in PCS.
≫show details for SCs
Once you and
your co-SCs have your list of conflicts, you should email this list to your ACs,
and
remind them to keep it handy: they should be checking it prior to any
communication
with you about specific papers. ACs cannot access this information through PCS.
- SCs identify papers that were initially assigned to their
subcommittee that should be
transferred to the secondary subcommittee chosen by the authors. As much as
possible,
we should honor authors' choice as much as we can, but for papers that's really
not
suitable, we can allow it to be transferred to a third subcommittee.
- Paper chairs run the plagiarism detection routine to identify any paper with
plagiarism problems, and pass the results to SCs to double check as candidates for
desk-rejects.
- ACs bid on papers assigned to their subcommittee in PCS.
≫show details for ACs
To bid on submissions:
- Log in to PCS 2.0,
click "Reviews"
and click "Review (as a committee member)"
- Click "Bidding"
- For at least 15 papers, select "want"
- For at least 15 papers, select "willing"
- ACs select at least 15 papers as “want” and 15
papers as
“willing”, for a total of at least 30.
- ACs specify their conflicts in PCS.
≫show details for ACs
Declaring conflicts with papers is helpful, but we know you can't necessarily
find every
paper that you may be conflicted with, so don't spend too much time on
this. Please do
declare your own submissions as conflicts.
To declare your conflicts:
- Log in to PCS 2.0
- Click "Reviews" at the top of your home page
- Click "Review (as a committee member)"
- Click "Conflicts"
Search institution with which you have conflict and select "conflict"
The ACM
conflicts policy is here.
|
|
Before Saturday, 18 Sep 2021
|
- SCs handle AC rebalancing issues based on feedback from
Papers Chairs.
≫show details for SCs
After papers are in we will let you know if you are over or
under on the number of ACs you need. The number of ACs we told you to recruit
was
based on projections from the last few years, but things may vary. If you have
more
ACs than needed then you can let someone off the hook. If you need more, we will
tell
you how many.
- SCs identify any papers that should be in a different
subcommittee and flag to papers
chairs.
≫show details for SCs
If you find a paper that you feel your
subcommittee does not have the right expertise to review, but another
subcommittee
does, please contact us ( papers@chi2021.acm.org) so we can
do
the switch. This should be relatively rare; if it's close to 50/50, even if you
would
have rather seen a paper submitted to a different subcommittee, we should
respect the
authors' choice.
- SCs for those committees with splits, split papers across
their subcommittee splits
and email papers chairs about splits’ composition if changed for rebalancing.
- SCs declare conflicts in PCS and pass responsibility to
co-SCs.
≫show details for SCs
You will find a ‘Declare chair conflicts’ link on your main page.
You only need
to declare conflicts with papers in your subcommittee.
Ideally you
will be able to
pass over the list of conflict papers to your co-chair. This happens by
email; there is no
formal mechanism for “owning” a paper as an SC; the point is
just to make sure
that you pass a list of conflicts along to ACs, so if they have questions,
they know which
SC not to ask. If your co-chair is also conflicted, please talk to
us.
Policy for SC conflicts
Because there is one
level of
indirection between you and reviews or discussion of the papers, we use a
slightly
loosened definition for conflict of interest for SCs that avoids some of
the more tenuous
connections of the full
policy. For declaring conflicts inside the system we suggest you
declare "core
conflicts" only:
- Papers you (co-)author
- Papers with an author at the local branch of your institution
- Papers (co-)authored by current close collaborators
- Any paper you feel you either can't judge impartially, or otherwise
represents a
conflict you feel you really should declare.
So you only have to declare conflicts with local colleagues and your close
collaborators. Otherwise you won't be able to chair the meeting as you'll be outside all
the time!
|
|
Monday, 20 Sept 2021
|
- Papers chairs will perform auto-assignment based on bidding and conflict information
collected from ACs.
- Papers chairs will release the initial paper assignment results to SCs.
|
|
Monday, 20 - Wednesday, 22 Sept 2021
|
- SCs double check the paper assignment released by Papers
chairs, and perform
adjustment if needed
|
|
Wednesday, 22 Sep 2021
|
|
|
Thursday, 23 Sept 2021 - Thursday, 30 Sept 2021
|
- ACs look for conflicts and other difficulties, optionally
swap problem papers, notify
SCs.
≫show details for ACs
The first thing you should do when you log in to
PCS is click each paper to check for conflicts. If you're conflicted with a
paper,
please request for a swap to your SCs. If you are not conflicted with a paper,
please
proceed to assigning reviewers.
- SCs track swaps and ensure all papers have a 1AC and 2AC
assigned in the end.
- 1ACs allocate two high-quality external reviewers to each
paper (for batch release
later).
≫show details for ACs
All review requests will be going out in batch on September 30th. Therefore,
during the current
reviewer selection phase, you should not be communicating with reviewers.
That means
please don't email reviewers to say "hypothetically, are you reviewing CHI
papers this
year?". The only exception: if a paper really requires a reviewer way
outside the CHI
community (someone's novel input device needs a physicist reviewer, or
someone's
hospital-based study needs a clinician reviewer), it's OK to contact them
to tell them
what CHI is, communicate our expectations for reviewers, and ask if they
are up for
reviewing papers. If you don't need to tell them what CHI is, you shouldn't
contact them
at this point.
For papers for which you are the 1AC you select two
reviewers. To help
focus on the importance of this task, we are asking that you be prepared at
the PC meeting
to give an explanation to the rest of the subcommittee for why you picked
the reviewer you
picked. The goal is to have exactly two review requests in place for each
paper by the
release date of September 30th, so please do not invite more than two
reviewer for
each of your 1AC
assignments. However, you might want to keep your own list of potential
backup reviewers
for your assignments.
You are looking for reviewers who will make
substantive
insightful comments, and who have the perspective to evaluate how
interesting results are
and whether they are sufficiently relevant to (some part of) HCI as a
field. Your first
line of attack for finding good reviewers may be your own knowledge -- if
the paper is in
your area (or close to it), you may be able to directly think of a good
candidate that you
already know to be an expert. Another excellent way to identify potential
reviewers is to
consider authors of previously published results on the topic. You will
likely find some
of these publications in the references of the paper itself. Searching for
related work in
the ACM Digital Library and other search
engines is also
typically very helpful. Keep in mind that publishing a single paper on a
topic, even at a
competitive venue such as CHI, might not mean a person is an expert, and
that different
authors may have contributed different things to a particular paper. For
potential
reviewers you are not familiar with in advance, it can be helpful to try to
have a look at
their overall research record through their web presence.
To actually make an assignment:
- Go to your reviewing home
- Click on the name of the paper
- Type a name (or email address) into the "Assign reviewer" field
- If there are multiple matches, click the one you want
- ACs identify papers as desk/quick reject candidates and
start negotiating with the
other AC - and if you have identified an early reject paper please notify SCs with a
short paragraph describing reasons for desk/quick reject See more here.
≫show details for ACs
Now you'll start skimming through your papers to select reviewers,
starting with your 1AC papers.
Also look for severe anonymization
issues.
We think we've handled most of these, but if you find any, notify your SCs right
away.
And, in your first pass through a paper, look for clear candidates
for
"desk rejection", i.e. rejecting papers without recruiting external reviewers.
We
expect that catastrophically bad writing will be the most common reason for desk
rejection. It's very hard for us to lay out objective criteria, but
subjectively,
it's "there is a zero percent chance that this paper will get in, and you would
not
feel comfortable using a qualified reviewer's time to handle it when there are
some
many papers to review."
With those identified, communicate and discuss with
other AC assigned and come to an agreement. Once agreed, email your SCs to
identify
these papers, and write up short rationale referring to the desk or quick reject
criteria and communicate to SCs (be careful about conflicts). The SCs will look
at
all the DR/QR candidates, and make a decision themselves. Once all ACs and SCs
agree
with the DR/QR decisions. The SCs email the paper chairs about their
recommendation.
All Quick rejected papers will have 4 people (ACs and SCs) looking at them, and
decisions will be recorded in PCS.
Minor formatting issues, and even (for
example) super-egregious reference formatting violations, are not grounds for
desk
rejection. Papers that aren't vaguely in 1 column ACM format are candidates for
desk
rejection.
- SCs review desk/quick reject candidates, and make
decisions.
≫show details for SCs
When scanning papers, please keep an eye out for submissions that have major
problems (e.g., ½ page in length, broken fonts, gross formatting
violations,
etc.) or otherwise have no chance of being accepted.
If the 1AC or 2AC
identifies a candidate for desk rejection, they should first communicate their
desk
reject recommendations to their SCs by email. Both the 1AC and 2AC will read the
paper. When a paper is raised for desk rejection, it will go to both the SCs and
the
papers chairs (with appropriate handling of chair conflicts).
|
|
Thursday, 30 Sept 2021
|
- Batch release of review requests to reviewers. We will release all of the review
requests to reviewers at the same time, to give ACs a
fair chance of getting who they
want as reviewers.
|
|
Thursday, 30 Sept 2021 - Thursday, 7 Oct 2021
|
- SCs calibrate desk/quick reject candidates. When in doubt,
SCs can email the paper
chairs to discuss any early reject candidates
|
|
Thursday, 14 Oct 2021
|
- Hard deadline for all desk/quick reject reviews (after SC calibration). If the review
is not ready or not in good enough quality, the quick reject decision is reverted,
and ACs need to find reviewers for the quick-reject paper
candidates.
|
|
Friday, 15 Oct 2021
|
- All desk/quick reject notifications sent out to authors.
|
|
Friday, 15 Oct 2021
|
- SCs (and/or their SC assistant) should go over all of the
technology with their ACs
|
|
Thursday, 30 Sept 2021 - Thursday, 28 Oct 2021
|
- ACs (and SCs) ensure that each paper gets two external
high-quality reviews: track
progress and find replacements for reviewers with the shortest delays as
possible.
≫show details for SCs
SCs need to take a particular concern for monitoring and encouraging
consistently high
quality across the review process, e.g., checking that the reviewer
selections are experts
and not just past students or friends, that there are (only) two reviewers,
etc. We ask
that SCs particularly mentor new/young ACs. We will leave the details of
how you do this
to SCs, but we can provide help if required.
You might want to ping
ACs who haven't
gotten all their reviewers lined up in a reasonable time for a status
report, as a means
to make sure they are working on it. We really need 2 high-quality external
reviews plus
the 2AC review for every paper going into rebuttal.
To monitor the
requests on the
Submissions page:
- Sort by the "Reviews total" column to see which submission do not have
enough reviews.
The "Reviews total" include reviewer and committee member reviews.
- Sort by the "Reviews tentative" column to see which submissions still
have
tentatively-assigned reviewers (i.e. reviewers who have not yet agreed
to review).
You can prompt the AC of each such submission to follow up with the
tentative
reviewers. Or you can send a reminder email to all tentative reviewers
from the Send
Email page.
- Sort by the "Reviews left" column to see which submissions still have
unsubmitted
reviews. These include the committee member reviews.
To monitor the requests on the Committee page:
- Sort by the "Reviews Remaining" column to see which committee members
have many
reviews left to do. Incomplete reviews are also shown as yellow boxes
in the "R" (for
"Review") columns on the right. You can send email to particular
committee members by
clicking on their name, then on their email address on the next page.
Early in this period, we recommend that you remind yourself and your
ACs of the
following guidelines that we use to communicate our standards to the CHI
community:
≫show details for
ACs
Until all reviews are in, the 1AC is responsible for
the both
external reviewers he/she assigned. As the review process
unfolds, you should
track your outstanding reviews for the papers you are in charge, send a
personal reminder
about a week before the deadline, and read over the ones you have
received. After a few days, check on reviewers who remain as
“tentative” in PCS, i.e., reviewers who have not clicked
“accept”
or “decline” yet. In your list of review assignments, you
should see a
Revs column which shows 4 numbers
(#completed/#assigned+#tentative+#stored). #tentative
shows the number of
pending review requests. If it looks like you may have lost a
reviewer, either
because you get an explicit notification from PCS, an email from the
reviewer, or any
other indication that the reviewer is declining or is unavailable (e.g.,
time has passed
with no communication at all), you should recruit a replacement review. You
can do this
directly through PCS, and there is no batch release process for replacement
reviewers
(your requests will go out immediately). At this stage, it is OK
to contact
potential reviewers by email. Monitoring and encouraging quality
reviews is the
second critical part of your job once reviewers are recruited. The most
common complaints
we receive from authors are that review decisions are not well justified or
that reviewers
have not used an appropriate tone or language. If you find reviews that you
think are of
lower quality than you would be proud to stand up and defend at the PC
meeting, you may
want to (very gently and diplomatically) suggest to them that they e.g.,
extend the
rationale given for their score, alter the tone or language they use, etc.
This is
important to ensure that all judgments are well justified, authors can
understand them,
and they can be weighed against opposing views. You might want to
remind yourself and
your reviewers of things like:
|
|
Friday, 29 October 2021
Note: There are two types of 3AC — 3AC by score & 3AC by request
|
- Reviews due back from reviewers and 2AC.
- 1ACs (and SCs) track missing
reviews & quickly resolve
(possibly with replacement
reviews).
≫show details for SCs
Track the status of papers and ensure that
no ACs have a pattern of problems (missing reviews). Prod ACs for status on
replacement reviews if needed. Again, it is critical that we have 2 high-quality
external reviews plus the high-quality 2AC review for every paper going into the
rebuttal process.
≫show details for
ACs
Quickly resolve
missing
reviews. Track missing reviews & quickly resolve issues
with
reviewers you that have recruited as 1AC or 2AC. In rare cases it may be
necessary to
find last-minute replacement reviews. It is critical that we have two
high-quality
external reviews for every paper going into rebuttal.
- ACs initiate reviewer discussion as needed.
≫show details for ACs
Use the
PCS reviewer discussion system to encourage discussion if the reviewer opinions
on a
paper are split. This is meant to help you formulate a thorough meta-review that
appropriately captures reviewer opinion, and also to make sure that no reviewer
missed or misunderstood something that another reviewer is able to clarify. The
discussion period closes on 10 Nov.
|
|
Saturday, 30 October 2021
|
- Papers chairs assign preliminary “discuss” status based on scores,
indicating which papers need a 3AC assigned. Note that there are two types of 3ACs.
The orange type is the 3AC assigned based on score criteria, which is done before the
rebuttal. The yellow type is the 3AC assigned after the 1AC’s meta-review,
which will enter their review after the rebuttal due to time constraints. They have
different deadlines in completing their reviews.
≫show details for SCs
SCs assign a 3AC to all papers that have a discuss status, being sure
to balance
the 3AC load across all of your ACs.
≫show details for ACs
The Papers
Chairs will determine a preliminary “discuss” status for papers
(i.e.,
papers that may be accepted, and therefore discussed at the PC meeting), based
on
numeric criteria (we’ll let you know what these criteria are on 5 Nov).
Papers
with a “discuss” status will get an additional meta-review from a
newly
assigned 3AC (deadline: Nov. 26). Papers without a “discuss” status
will
only get a meta-review from the 1AC.
-
Online discussions start.
≫show details for SCs
SCs go through all submissions and identify submissions where recommendations among reviewers are split.
They can be 3AC candidates nominated by SCs.
≫show details for ACs
1AC should immediately start online discussions on each submission in their pile.
1AC reads the reviews and considers if they would like to have a 3AC to reach a consensus.
|
|
Saturday, 30 Oct 2021 - Wednesday, 3 Nov 2021
|
- ACs ensure to facilitate a discussion bet
- Discussion between reviewers
|
|
Monday, 1 Nov 2021
|
1AC must tick the corresponding box in the review field to request a 3AC.
|
|
Tuesday, 2 Nov 2021
|
SCs assign 3ACs for those requested papers as well as papers that they think would be valuable to have 3ACs.
|
|
Wednesday, 3 Nov 2021
|
3AC joins online discussions after reading the assigned papers. They may submit their own reviews if they wish.
1AC keeps actively running online discussions to seek consensus on the recommendation for each submission.
1AC completes their meta reviews and indicates preliminary recommendations of Accept with minor revision, Revise and Resubmit, and Reject.
|
|
Thursday, 4 Nov 2021
|
|
|
Thursday, 4 Nov 2021 - Thursday, 11 Nov 2021
|
- 1AC completes their meta reviews and indicates preliminary recommendations of Accept
with minor revision, Revise and Resubmit, and Reject.
|
|
Thursday, 11 Nov 2021 - Saturday, 13 Nov 2021
|
|
|
Sunday 14 Nov 2021 - Wednesday, 17 Nov 2021
|
- AC1 finalises meta-review, based upon own reading, external reviewers, AC2 review and
potential AC3 review
|
|
Sunday, 14 Nov 2021 - Wednesday, 17 Nov 2021
|
- SCs checks the quality of the meta reviews
|
|
Thursday, 18 Nov 2021
|
- Reviews released to Authors
|
|
Thursday, 2 Dec 2021 (TBC)
|
- Publication Ready Deadline (1st Round)
|
|
Monday, 10 Jan 2022
|
- Final Publication Proofs Due (1st Round)
- Resubmission Due from Authors
|
|
Friday, 14 Jan 2022
|
- ACs & SCs check
resubmissions. If authors did not
provide response letter or
highlight submissions, SCs must communicate with these
authors and get resubmission
in 24 hours
- 1AC starts looking at A1 pub ready (this is aligned with the date for reviewers to start re-review)
|
|
Friday, 14 Jan 2022 - Monday, 31 Jan 2022
|
- 1AC asks original reviews to re-review OR recruit a new reviewer as appropriate
|
|
Friday, 28 Jan 2022
|
|
|
Tuesday, 1 Feb 2022
|
|
|
Wednesday, 2 Feb 2022 - Saturday, 5 Feb 2022
|
- Online discussions among reviewers, 1AC submits meta reviews.
|
|
Sunday, 6 Feb 2022 - Monday, 7 Feb 2022
|
|
|
Tuesday, 8 Feb 2022 - Wednesday, 9 Feb 2022
|
- AC finalizes meta reviews, clearly enumerate the points that need to be fixed for
final
|
|
Thursday, 10 Feb 2022
|
- Reviews released to authors
|
|
Thursday, 24 Feb 2022 (TBC)
|
- Publication Ready Deadline
|
|
Thursday, 25 March 2022 (TBC)
|
- Publication Ready Proofs Approved
|
