Call for Reproducibility Papers

ECIR strongly encourages the submission of papers that repeat, reproduce, generalise, and analyse prior work. Please refer to the ACM "Artifact Review and Badging" guidelines for consistent use of the terminology, which is heterogeneous across disciplines.

Reproducibility is key for establishing research to be reliable, referenceable, and extensible for the future. Emphasise your motivation for selecting the paper/papers, the process of how results have been attempted to be reproduced (successful or not), the communication that was necessary to gather all information, the potential difficulties encountered, and the result of the process. A successful reproduction of the work is not a requirement, but it is crucial to provide a precise and rigid evaluation of the process to allow lessons to be learned for the future.

Topics of Interest

We welcome submissions on any topic relevant to the general field of Information Retrieval, including those mentioned in the Call for Full papers for ECIR 2027.

Note: In particular, we solicit replicability (different team, same experimental setup) and reproducibility (different team, different experimental setup) papers. Submissions from the same authors – i.e., repeatability (same team, same experimental setup) papers – of the reproduced experiments will not be accepted.

Review Criteria

All reproducibility-track papers will be evaluated along the following criteria (when applicable):

Reliability

• Is the evaluation methodology in line with the research challenges addressed by the reproduced experiment?
• Are the selected baselines representative of the several algorithm types and techniques available?
• Is the parameter/hyperparameter setting properly described?
• Are algorithms and baselines adequately tuned?
• Does the study attempt to replicate appropriate portions of the original paper?

Impact

• How important is the reproduction of the experiments to the community?
• How obvious are the conclusions achieved?
• Do the reproduced prior works, if validated, advance a central topic to information retrieval (a topic with broad applicability or focused on a hot research area)?
• What is the impact of the original paper? Is it central or marginal to the community?

Novelty

• Are there new insights or perspectives that emerged from reproduction which were not reported in the original paper?
• Are there new baselines and experiments presented in the reproduced paper?
• Is the reproduced paper proposing new evaluation criteria (new measures, statistical tests, etc.)?

Availability

• Are the code and datasets used to reproduce the experiments available to the reviewers at the time of review?
• Is the shared material released in a permanent repository for easy access by researchers?
• Are the reproduced experiments well documented, with all the details required for other researchers to reproduce the experiments?
• Are there discrepancies between what is described in the paper and what is available in the shared material?
• Is the shared material complete with everything you need to replicate the experiments exactly?

Submission Guidelines

• Reproducibility papers can be up to 12 pages in length plus additional pages for references. Appendices count toward the page limit.
• Reproducibility papers will be refereed through double-blind peer review, with an initial first stage review followed with a second stage of discussion led by a meta-reviewer.
• Authors are asked to upload their software artifacts either (i) to an anonymous repository that is linked to in the submission, such as an anonymous github, or (ii) to Easychair as part of the submission.

Authors should consult Springer's author guidelines and use their proceedings templates, either for LaTeX or Word for the preparation of their papers. The templates are available at https://bit.ly/springer-guidelines. Springer encourages authors to include ORCIDs in their papers (https://www.springer.com/gp/authors-editors/orcid). 

All submissions must be written in English. All papers should be submitted electronically through the EventsAir submission system: here

In addition, the corresponding author of each accepted paper, acting on behalf of all of the authors of that paper, must complete and sign a Consent-to-Publish form. The corresponding author signing the copyright form should match the corresponding author marked on the paper. Once the paper has been submitted, changes relating to its authorship cannot be made.

Accepted papers will be published in the conference proceedings in the Springer Lecture Notes in Computer Science series. The proceedings will be distributed to all delegates at the conference. Accepted papers will have to be presented at the conference by one of the authors in person – and at least one author for each accepted contribution will be required to register and attend.

Dual Submission Policy

Papers submitted to ECIR 2027 should be substantially different from papers that have been previously published, or accepted for publication, or that are under review at other venues. Exceptions to this rule are:

• Submission is permitted for papers presented or to be presented at conferences or workshops without proceedings.
• Submission is permitted for papers that have previously been made available as a technical report (e.g., in institutional archives or preprint archives like arXiv). However, we discourage this since it places anonymity at risk; in particular, please do not publish your paper at arXiv and submit it to ECIR at the same time, some days before, or during the reviewing period of ECIR. If your paper is already available as a technical report, you might not want to use the exact same title and abstract for your ECIR submission (in case of acceptance at ECIR, the title of your submission still might be changed back).
• Please do not cite your technical report and make some effort to avoid any issues that may harm the double-blindness of your submission. Reviewers will receive guidance that ask them to refrain from trying to break blindness if at all possible too, but be aware that the availability of an available technical report for an ECIR submission can cause issues.