mlsanityData to 7 Oct 2026

Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement

arXiv 2606.22166 ↗First posted 20 Jun 2026cs.DL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Author rebuttals are the main post-submission window in peer review, but their effect on reviewer scores remains hard to measure because score updates mix rebuttal content with initial score position, paper-level consensus, reviewer confidence, and discussion dynamics. We study ICLR 2024-2025 using 73,000 reviewer trajectories with externally archived pre- and post-rebuttal scores, and use LLMs only as measurement instruments. Gemini Flash 3.0 predicts implied pre-rebuttal scores from score-stripped review text. The resulting text-score offset predicts later movement, with score-increase rates rising from 8.3% when text reads below the assigned score to 31.9% when it reads above. Claude Opus 4.6 induces, and outcome-blinded Gemini Flash 3.0 validates, a 44-feature taxonomy of resolved reviewer-author exchanges, where 23 features replicate across model and held-out year under Bonferroni correction. In the rebuttal-engaged benchmark (n=6,705), initial-review structure already predicts much score movement (AUC=0.747, minimal AUC=0.696), while adding the resolved exchange raises AUC to 0.804. Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

27 of 46 references.

References · 27

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  3. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  4. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  5. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  6. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  7. 2026Towards End-to-End Automation of AI Researchcited by 150
  8. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 135
  9. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
  10. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 66
  11. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  12. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
15 more
  1. 2019Argument Mining for Understanding Peer Reviewscited by 41
  2. 2021DISAPERE: A Dataset for Discourse Structure in Peer Review Discussionscited by 35
  3. 2019Does My Rebuttal Matter? Insights from a Major NLP Conferencecited by 31
  4. 2023Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experimentcited by 31
  5. 2026Scientific production in the era of Large Language Modelscited by 30
  6. 2023Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experimentscited by 26
  7. 2020An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Processcited by 23
  8. 2025Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Reviewcited by 18
  9. 2024Automating the Practice of Science -- Opportunities, Challenges, and Implicationscited by 17
  10. 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 12
  11. 2023What makes a successful rebuttal in computer science conferences? : A perspective on social interactioncited by 10
  12. 2025Paper Copilot: Tracking the Evolution of Peer Review in AI Conferencescited by 10
  13. 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
  14. 2026Defend: Automated Rebuttals for Peer Review with Minimal Author Guidancecited by 3
  15. 2026Decoupling Scores and Text: The Politeness Principle in Peer Reviewcited by 1

Built on by

Not cited yet.