# CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers

arXiv: https://arxiv.org/abs/2605.07905 · HTML: https://mlsanity.com/paper/2605.07905/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 74 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 27 of 27)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding](https://mlsanity.com/paper/2503.05061/) (2025)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025](https://mlsanity.com/paper/2504.09737/) (2025)
- [Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions](https://mlsanity.com/paper/2406.05688/) (2024)
- [ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews](https://mlsanity.com/paper/2503.08506/) (2025)
- [Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments](https://mlsanity.com/paper/2311.09497/) (2023)
- [ReviewEval: An Evaluation Framework for AI-Generated Reviews](https://mlsanity.com/paper/2502.11736/) (2025)
- [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025)
- [When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review](https://mlsanity.com/paper/2509.09912/) (2025)
- [REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2505.11718/) (2025)
- [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2505.16176/) (2025)
- [Reviewer2: Optimizing Review Generation Through Prompt Generation](https://mlsanity.com/paper/2402.10886/) (2024)
- [REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning](https://mlsanity.com/paper/2505.19862/) (2025)
- [AI and the Future of Academic Peer Review](https://mlsanity.com/paper/2509.14189/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review](https://mlsanity.com/paper/2609.05947/) (2026)
- [IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation](https://mlsanity.com/paper/2610.04074/) (2026)

## Abstract (arXiv)

Despite the rapid development of AI reviewers, evaluating such systems remains challenging: metrics favor overlap with human reviews over correctness. However, since human reviews often cover only a subset of salient issues and sometimes contain mistakes, they are unreliable as gold references. To address this, we build category-specific benchmark subsets and skip evaluation when the corresponding human reviews are missing to strengthen Completeness. We also leverage reviewer--author--meta-review discussions as expert annotations and filter unreliable reviews accordingly to strengthen Correctness. Finally, we introduce CoCoReviewBench, which curates 3,900 papers from ICLR and NeurIPS to enable reliable and fine-grained evaluation of AI reviewers. Analysis shows that AI reviewers remain limited in correctness and are prone to hallucinations, and highlights reasoning models as more effective reviewers, motivating further directions for improving AI reviewers. Benchmarks and models are available at https://github.com/hexuandeng/CoCoReviewBench.