More Criticism Does Not Make a Better Review: EquiReview-R
Not on the map. Why?
Abstract (arXiv)
AI reviewers can now produce many specific criticisms, but more criticism is not necessarily a better review. A review may miss a consequential weakness or retain an allegation that available evidence does not support. These failures require opposite corrections, yet generation-oriented systems and aggregate measures obscure the distinction. We therefore recast AI-assisted review as evidence-guided refinement of a structured concern set, with omission and overcritique treated as separate risks. Building on this formulation, we introduce EquiReview-R, which resolves existing concerns against localized evidence, searches for missing issues from independent and review-conditioned perspectives, and returns stop, continue, or defer. To expose the failure mode that motivates this design, we construct an evidence-linked trajectory corpus. Its retrospective analysis shows why revision must precede further search: nearly all concerns in a high-recall review lack a definitive evidential disposition, while an earlier refinement mechanism cannot revise them. On a frozen cohort of previously unseen papers, EquiReview-R satisfies the prespecified non-inferiority criterion for major omission, reduces major overcritique from 15.5% to 8.1%, and attains a one-sided omission upper bound of 9.9% while stopping on 52.4% of papers. Computation-matched controls, controlled pairs, and ablations show that the gain comes from revision rather than extra inference or shorter output. We release the corpus as ReviewTrace, an evidence-linked resource for studying review revision, disagreement, and provenance.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 3 Sep 2026.
Builds on
23 of 29 references.
References · 23
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
- 2022Self-critiquing models for assisting human evaluatorscited by 380
- 2022Conformal Risk Controlcited by 299
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
- 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 164
- 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
- 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
- 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
- 2024Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Reviewcited by 53
- 2021MReD: A Meta-Review Dataset for Structure-Controllable Text Generationcited by 42
11 more
- 2023Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experimentcited by 34
- 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
- 2025Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviewscited by 24
- 2026Stop Automating Peer Review Without Rigorous Evaluationcited by 14
- 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
- 2026DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoningcited by 6
- 2026On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientistscited by 6
- 2026SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Promptscited by 4
- 2026From Passive Generation to Investigation: A Proactive Scientific Peer Review Agentcited by 4
- 2026What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Reviewcited by 2
- 2026E3: Issue-Level Backtesting for Automated Research Critiquecited by 2
Built on by
Not cited yet.