Senior PC members initiate and guide reviewer discussions and write meta-reviews for papers that provide crucial information to make the final decisions about acceptance for the EDM 2021 conference. We very much appreciate the time and attention that you give to these responsibilities.
Guidelines during reviewing period:
If you’ve completed your review and you notice that any of the other reviews do not meet the reviewing guidelines, please reach out to that reviewer via EasyChair and ask them to update their review, ideally pointing out what about their review is lacking. Additionally, if it is close to the review deadline and you notice that some reviews are missing on those papers, please also reach out to the reviewer.
Guidelines for discussions:
Immediately after the reviewing period has ended, please check that each of the papers for which you have been assigned to be the metareviewer has three reviews (including your own) and that these reviews meet the reviewing guidelines, including being substantive and constructive. If any of the reviews do not meet these criteria, please send a message to the reviewer through EasyChair and ask them to provide or update their review; it’s often helpful to be specific about how the review is inconsistent with the reviewing guidelines. If your efforts to contact the reviewer do not lead to changes to the review, please notify the PC Chairs (Sharon Hsiao & Sherry Sahebi).
(Note that there may be some papers that you reviewed but for which you are not the metareviewer. You should participate in discussions about those papers, but you do not need to write the metareview or proactively reach out to other reviewers.)
If all of your papers have appropriate reviews, please examine whether the reviewers are generally in agreement. If they are in agreement, you might write the metareview (see below for guidelines) and then initiate a discussion in which you ask reviewers whether they agree with the metareview or have anything they feel was not well represented. If the reviewers are not in agreement, then please initiate a discussion on EasyChair among the reviewers to try to work out disagreements and come to some consensus. It is not necessary that all reviewers agree about their numeric ranking, but discussion of large differences in ranking or disagreements about the claims or merit of a paper can uncover and address misunderstandings.
Guidelines for metareviews:
Based on your own review and the reviews from the other PC members, you should write a metareview that provides an overall recommendation for Accept or Reject (including the strength of that recommendation) and summarizing the main reasons for that recommendation, drawing on the points that were common across reviews and most important in any discussion of the paper among reviewers. Your meta-review should be about a paragraph long and should be sure to draw on all of the reviews, not only your own. You should use your judgment to decide on the relative importance of some of the points brought up by you and other reviewers; if some reviewers believe they have less expertise, this should be considered in evaluating their comments.
In some cases, the reviews or discussion may suggest that the paper should be accepted as something other than the category in which it was submitted. For example, the reviewers might suggest that the contribution is not large enough for a full paper, but that the methodology and analyses are sound and thus that a short paper (or poster) would be more appropriate. We cannot change the EasyChair metareview page to collect this information from a multiple choice selection. Instead, in the confidential information to program committee box on the metareview page, please indicate one of the following recommendations, based on taking the totality of the reviews into account:
- Recommend acceptance in submitted category
- Recommend acceptance as a short paper instead of submitted category
- Recommend acceptance as a poster paper instead of submitted category
- Recommend rejection
You might recommend acceptance in a different format due to concerns on the part of the reviewers about the size of the contribution or about the work being less fleshed out. However, if the reviewers agree that there are significant issues in the accuracy or methodology, that likely suggests a rejection rather than acceptance in another form.