We are committed to setting a high standard of quality and integrity for the EDM2022 review process. There are two main goals in reviewing: (1) To decide fairly whether each paper is worthy of acceptance; and (2) To provide the authors with feedback on how to improve the quality of their research and writing. With these goals in mind, we request that all reviewers follow the review guidelines (based loosely on the ICLR2020 guidelines) shown below:
1. Read each of your assigned papers at least 2x: Understanding clearly the paper’s merits and weaknesses is paramount to good reviewing. Oftentimes, aspects of the paper that were unclear during the first pass are much clearer on the second pass. Similarly, gaps in the paper’s methodology can sometimes be identified more definitively during a second reading.
EDM 2022 incorporates questions related to fairness, equity, and positive social impacts as part of the review process this year. Authors should thus also consider these questions while writing papers. Details are available online (https://educationaldatamining.org/edm-2022-author-reviewer-checklist/); note that not all questions will be relevant to every paper, but if a question is relevant, you should pay attention to it.
2. Answer four key questions for yourself, to make a recommendation for Accept or Reject:
- What is the specific question/problem tackled by the paper? This is important to the authors so they can be certain that their main points came through clearly. It is also important to the Senior Program Committee, when writing meta-reviews and deciding on the final Accept/Reject decision, to ensure that the reviewer understood the paper.
- Is the approach well motivated, including being well-placed in the literature?
- Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
- Does the paper mention issues regarding fairness, equity, and positive social impacts?
3. Organize your review as follows:
- Begin by summarizing what the paper claims to do/contribute. Be positive and generous. (This should typically be its own paragraph.)
- Clearly state your recommendation (accept or reject) with a short, high-level justification (e.g., “While the paper presented some novel ideas, there was not enough empirical validation.”).
- Provide supporting arguments for the reasons for the decision (e.g., “Section XYZ is missing an important analysis that would be needed to support claim ABC.”)
- Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
- Provide comments on relevant questions regarding fairness, equity, and positive social impacts.
Frequently Asked Questions
What should I do if the paper is not properly anonymized?
Please indicate your concerns in the “confidential information to program committee” box on the reviewing form, but otherwise, review the paper as if it were properly anonymized.
If I’m sure the paper should definitely be accepted (or be rejected), do I still need to include all four parts of the review?
Yes! Complete reviews, which are generally no less than several paragraphs long, are important for all papers. An “instant accept” for one reviewer may not be viewed the same way by another reviewer, and without a complete review, it’s impossible to know if that disagreement is due to a misunderstanding, a different lens into the methodology, or some other cause. Understanding the reason for differences across reviews allows for better final decisions about the papers. Additionally, reviews provide valuable information for authors to improve their work. Please remember that all authors are trying to provide high-quality work that is appropriate for EDM, and providing constructive feedback may help them improve their contributions to the field in the future.
What should I write if the paper is missing important citations?
Please provide in your review an itemized list of at least 1-2 references that are clearly missing from the paper.
How should poorly-written English affect the review?
It is completely legitimate to give a paper a low score if the quality of the writing prevents you from understanding the paper if the writing is so unclear that the reviewer is not confident that experiments and analyses were performed correctly, etc. However, a paper can still make a strong scientific contribution and be accepted even if its English is not perfect or is written by a non-native speaker! Grammatical mistakes, misspellings, and unusual phrasing should not affect the reviewer’s score as long as the paper is scientifically sound, well-motivated, etc. On the other hand, it can be very helpful to the authors if you specifically note the English errors so that the authors can address them. Typically, this would be done at the end of your review; you may even wish to state explicitly (if appropriate) that these errors did not impact the review scores.
I think the paper has crucial methodological flaws. How specific do I need to be about the issue, given that I’m not going to recommend it be accepted?
It’s important that you’re specific about the methodological flaws for two reasons: (1) it helps the other reviewers understand where you’re coming from, including potentially convincing them to change their review and recommendation, and (2) it helps the authors to address these flaws in future submissions, leading to higher quality work overall. Please be clear about what the methodological issue is, and if relevant, note and briefly explain what dataset was not evaluated on but should have been; what experiment should be conducted to clarify the results; what baseline condition you would have liked to have seen; etc. In some cases, you may be noting an issue with the use of statistics or evaluation metrics; again, the more specific you can be about what is missing or wrong, the more helpful your review will be.
The paper I’m reviewing seems to make a valid scientific contribution but is not very novel. How much novelty is required for me to recommend the paper for acceptance?
EDM 2022 explicitly welcomes replication and reproductions of previous work, with the goal of contributing to a solid scientific foundation on which everyone can build. All papers should be well-grounded in the literature, and lack of reference to the paper that is being replicated would of course be grounds for a recommendation to reject. Novelty is one characteristic to consider when evaluating a paper; methodological soundness and scientific rigor are two other important characteristics.