Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement
Not on the map. Why?
Abstract (arXiv)
Author rebuttals are the main post-submission window in peer review, but their effect on reviewer scores remains hard to measure because score updates mix rebuttal content with initial score position, paper-level consensus, reviewer confidence, and discussion dynamics. We study ICLR 2024-2025 using 73,000 reviewer trajectories with externally archived pre- and post-rebuttal scores, and use LLMs only as measurement instruments. Gemini Flash 3.0 predicts implied pre-rebuttal scores from score-stripped review text. The resulting text-score offset predicts later movement, with score-increase rates rising from 8.3% when text reads below the assigned score to 31.9% when it reads above. Claude Opus 4.6 induces, and outcome-blinded Gemini Flash 3.0 validates, a 44-feature taxonomy of resolved reviewer-author exchanges, where 23 features replicate across model and held-out year under Bonferroni correction. In the rebuttal-engaged benchmark (n=6,705), initial-review structure already predicts much score movement (AUC=0.747, minimal AUC=0.696), while adding the resolved exchange raises AUC to 0.804. Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 20 Jun 2026.
Builds on
27 of 46 references.
References · 27
- 2018Datasheets for Datasetscited by 1,371
- 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
- 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
- 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
- 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
- 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
- 2026Towards End-to-End Automation of AI Researchcited by 150
- 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 135
- 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
- 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 66
- 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
- 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
15 more
- 2019Argument Mining for Understanding Peer Reviewscited by 41
- 2021DISAPERE: A Dataset for Discourse Structure in Peer Review Discussionscited by 35
- 2019Does My Rebuttal Matter? Insights from a Major NLP Conferencecited by 31
- 2023Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experimentcited by 31
- 2026Scientific production in the era of Large Language Modelscited by 30
- 2023Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experimentscited by 26
- 2020An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Processcited by 23
- 2025Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Reviewcited by 18
- 2024Automating the Practice of Science -- Opportunities, Challenges, and Implicationscited by 17
- 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 12
- 2023What makes a successful rebuttal in computer science conferences? : A perspective on social interactioncited by 10
- 2025Paper Copilot: Tracking the Evolution of Peer Review in AI Conferencescited by 10
- 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
- 2026Defend: Automated Rebuttals for Peer Review with Minimal Author Guidancecited by 3
- 2026Decoupling Scores and Text: The Politeness Principle in Peer Reviewcited by 1
Built on by
Not cited yet.