Content and track choice for submissions

What is the difference between the different types of submissions?

Full, short, industry, poster, and demo paper submissions should all describe high quality, previously unpublished work. Full papers typically represent larger contributions, perhaps with more thorough evaluation and a wider breadth. Full papers may be up to 10 pages in length, plus references. Short papers may be up to 6 pages in length, plus references. 

Industry submissions may be up to 6 pages in length, plus references, and should also represent new research, conducted in an industry context. Work conducted in industry may also be submitted to any of the other tracks. “Industry” can include for-profit companies and corporations (e.g., Canvas), as well as non-profit or academic institutions (e.g., edX.org) that operate at a large scale. The industry track specifically highlights work where the industry context is important, e.g., how learning analytics can be scaled to millions of users/customers, or examining discrepancies between expected and actual results of an intervention specific to a particular platform.

Poster submissions often present late-breaking work, or work that is still in-progress; in all cases, these submissions should be self-contained and explain a complete story. These submissions may be up to 4 pages in length, plus references. Demo papers are appropriate for disseminating information about and showing tools or systems to the conference audience in real time. Demo papers can be up to 4 pages (though shorter may also be appropriate).

Workshop and tutorial submissions should present a plan for a half- or full-day workshop or tutorial about an important and/or emerging area for EDM. The submission should describe the purpose of the workshop, expected format (including the relative weight of discussion, panels, talks, etc.), proposed invited speakers, and expected outcomes for attendees.

Doctoral consortium submissions are a great way for students to get feedback about their ongoing and proposed upcoming work. The doctoral consortium is intended to provide mentorship and will include opportunities to get input from the community about proposed research questions and methods. In addition to doctoral students, undergraduate and masters students who are interested in receiving mentorship and feedback on their ongoing research are also very welcome in the EDM 2020 doctoral consortium. The paper submission for the doctoral consortium should include both some information about the research that has been conducted so far, as well as a description of the planned future work. The research conducted so far that is described in the doctoral consortium submission may be research that is under review at EDM (and/or under review or published elsewhere). It should also include an explicit statement about what aspects of the research the submitter is seeking advice about.

Do you have any suggestions for how to make my paper as compelling as possible?

The most compelling papers make their unique research contributions very clear and provide sufficient details for the reader to be able to evaluate (and ideally reproduce) the methods and results. Here are some tips from past reviewers and authors at EDM:

  • For papers that introduce novel algorithms or data analysis methods, it is important to compare your results to existing methods, including current state-of-the-art approaches.
  • When proposing a complex method that includes several modular components (e.g., a convolutional neural network followed by a recurrent neural network followed by a clustering method), consider performing an ablation study that includes only a subset of these components. This can help to disentangle what is added by each part of the approach.
  • Papers that replicate prior studies, as well as those that “close-the-loop” and deploy known algorithms in unconstrained real-life settings, are also welcome. Such papers should explain why the result/method that is being tested is particularly important and provide a convincing argument that the replication efforts were rigorous. 
  • For analyses of both pre-existing datasets and new datasets or behavioral experiment results, include information about sample size (e.g., “n=26” for 26 participants).
  • For experimental results, specify how participants were recruited (e.g., from an undergraduate course at a large public university, or online on Amazon Mechanical Turk).
  • For estimating treatment effects, randomized-controlled trials (RCT) are arguably the gold standard and are encouraged. Papers that use non-random assignment or observational analyses can sometimes be strengthened by acknowledging potential confounds or discussing how they can be mitigated.
  • When presenting the results of statistical tests, consider including effect sizes rather than only p-values, and always include the values for test statistics and enough information for the reader/reviewer to know what test was conducted.
  • Use EDM-standard validation approaches. For example, perform cross-validation that is stratified over students or classrooms rather than observations. When characterizing classification/regression accuracy, use multiple metrics (e.g., RMSE, Pearson correlation for regression; F1 scores, ROC curves, and AUC scores in addition to just %-correct classification rate). If you use non-standard EDM validation approaches, then explicitly argue in favor of them.
  • In your discussion section, include limitations and “threats to validity” in addition to proposed future work.
  • Although not required for EDM 2020, your paper can be strengthened and its impact increased by pre-registering the research hypotheses and data analysis plan.
  • You might consider submitting your code (anonymously) as a link embedded in your paper; this can help promote open science.

Can I submit code?

Yes. Authors are encouraged, but not required, to submit a link to the code they used to implement models, run experiments, etc. This can help to promote open science. Note that the paper should still be self-contained for future readers, and reviewers will not be required to look at or evaluate the code. Additionally, make sure that all code is submitted anonymously — see section on double-blind reviewing below.

Can I submit supplemental materials or other content that does not count towards the page limitations?

While you may include a link to the code that you used in order to support open science, other links to supplemental material are not permitted. These materials cannot be considered during the reviewing process, and the paper should stand on its own without such materials.

Double blind review

Please note that the Double Blind Review process is used for the full, short, poster, demo, and doctoral consortium tracks. It is not used for workshop/tutorial or industry track submissions.

Why use double blind review?

Double blind review helps to avoid unconscious bias on the parts of reviewers and level the playing field for all authors. We therefore require submissions to be anonymized appropriately (see below) prior to submission. For some more information about double blind review in computer science, see Claire Le Goues’s blog post, a summary of Hanna Wallach’s ICML 2013 workshop talk, and Tomkins, Zhang, and Heavlin’s 2017 PNAS paper about bias with single-blind reviewing in a highly selective data mining conference.

In some cases, a reviewer might still have a guess about the authors’ identities based on certain details in the paper, e.g., a specific dataset or tool that was used that is accessible to only a few research labs. However, even in these cases, properly anonymizing the paper can still provide most of the benefits of double-blind review by casting doubt on who wrote the paper.

How should we prepare our paper for double-blind reviewing?

Your goal in preparing your paper is to make it easy for reviewers to read and review your paper without knowing the identity of the authors or their institution. You should do at least the following, with these suggestions lightly edited from ICSE 2019 and ASE 2018:

  1. Most importantly: Omit all authors’ names and institutions from the title page.
  2. Refer to your own work in the third-person. For instance, if the authors submitting the paper are Schmidt and Lopez, then instead of writing, “In prior work, we conducted an analysis of XYZ…”, they should write, “In prior work, Schmidt & Lopez (2016) conducted an analysis of XYZ…”
  3. If revealing the name of your tool or system reveals your or your institution’s identity, you should use an alias name for the tool/system in your submission. However, if the system or tool you’re working with is open-source and/or widely available for research, there is no need to rename the system. If you’re reporting new experimental results that use students or participants at a specific institution, please omit the name of the institution for the submission, giving only relevant details for understanding the context of the experiment (e.g., “students in a very large introductory CS course taught at a community college”). If you have any doubts, please contact the PC Chairs. 

The ACM also provides some great information about anonymization.