# Does My Rebuttal Matter? Insights from a Major NLP Conference

arXiv: https://arxiv.org/abs/1903.11367 · HTML: https://mlsanity.com/paper/1903.11367/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 30 of 31)

- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions](https://mlsanity.com/paper/2110.08520/) (2021)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [MOPRD: A multidisciplinary open peer review dataset](https://mlsanity.com/paper/2212.04972/) (2022)
- [What Can Natural Language Processing Do for Peer Review?](https://mlsanity.com/paper/2405.06563/) (2024)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals](https://mlsanity.com/paper/2311.03998/) (2023)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond](https://mlsanity.com/paper/2201.11443/) (2022)
- [Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision](https://mlsanity.com/paper/2406.00197/) (2024)
- [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025)
- [Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance](https://mlsanity.com/paper/2601.14171/) (2026)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge](https://mlsanity.com/paper/2603.07019/) (2026)
- [Some Ethical Issues in the Review Process of Machine Learning Conferences](https://mlsanity.com/paper/2106.00810/) (2021)
- [Did AI get more negative recently?](https://mlsanity.com/paper/2202.13610/) (2022)
- [The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics](https://mlsanity.com/paper/2312.04576/) (2023)
- [Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2505.15031/) (2025)
- [Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review](https://mlsanity.com/paper/2602.11173/) (2026)
- [Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective](https://mlsanity.com/paper/2109.01190/) (2021)
- [NLLG Quarterly arXiv Report 06/23: What are the most influential current AI Papers?](https://mlsanity.com/paper/2308.04889/) (2023)
- [What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review](https://mlsanity.com/paper/2604.19998/) (2026)
- [Decoupling Scores and Text: The Politeness Principle in Peer Review](https://mlsanity.com/paper/2604.14162/) (2026)
- [What do writing features tell us about AI papers?](https://mlsanity.com/paper/2107.06310/) (2021)
- [An Inclusive Notion of Text](https://mlsanity.com/paper/2211.05604/) (2022)
- [Which Coauthor Should I Nominate in My 99 ICLR Submissions? A Mathematical Analysis of the ICLR 2026 Reciprocal Reviewer Nomination Policy](https://mlsanity.com/paper/2512.17950/) (2025)
- [Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time](https://mlsanity.com/paper/2601.15172/) (2026)
- [Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement](https://mlsanity.com/paper/2606.22166/) (2026)

## Abstract (arXiv)

Peer review is a core element of the scientific process, particularly in conference-centered fields such as ML and NLP. However, only few studies have evaluated its properties empirically. Aiming to fill this gap, we present a corpus that contains over 4k reviews and 1.2k author responses from ACL-2018. We quantitatively and qualitatively assess the corpus. This includes a pilot study on paper weaknesses given by reviewers and on quality of author responses. We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i.e., final) scores from initial reviews and author responses. Although author responses do have a marginal (and statistically significant) influence on the final scores, especially for borderline papers, our results suggest that a reviewer's final score is largely determined by her initial score and the distance to the other reviewers' initial scores. In this context, we discuss the conformity bias inherent to peer reviewing, a bias that has largely been overlooked in previous research. We hope our analyses will help better assess the usefulness of the rebuttal phase in NLP conferences.