# PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing

arXiv: https://arxiv.org/abs/2605.29815 · HTML: https://mlsanity.com/paper/2605.29815/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 14 of 14)

- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [DeepSeek-OCR: Contexts Optical Compression](https://mlsanity.com/paper/2510.18234/) (2025)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions](https://mlsanity.com/paper/2406.05688/) (2024)
- [ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews](https://mlsanity.com/paper/2503.08506/) (2025)
- [ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature](https://mlsanity.com/paper/2403.16887/) (2024)
- [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025)
- [Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review](https://mlsanity.com/paper/2502.19614/) (2025)
- [ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review](https://mlsanity.com/paper/2510.08867/) (2025)
- [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025)
- [What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review](https://mlsanity.com/paper/2509.25701/) (2025)
- [Gen-Review: A Large-scale Dataset of AI-Generated (and Human-written) Peer Reviews](https://mlsanity.com/paper/2510.21192/) (2025)

## Built on by (most cited first, 2 of 2)

- [Local AI pre-screening for human triple-blind peer review in health sciences](https://mlsanity.com/paper/2608.14625/) (2026)
- [Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics](https://mlsanity.com/paper/2609.23264/) (2026)

## Abstract (arXiv)

The growing number of submitted papers has motivated the exploration of Large Language Models (LLMs) as a means to support and augment the peer review process, particularly in terms of improving its speed and scalability. Yet, it remains unknown whether LLMs engage with scientific manuscripts in the same manner as human reviewers, or whether they merely produce review-looking text. To address this, we introduce the Peer Review AI Benchmark (PRAIB), a novel framework comprising thoroughly defined metrics that measure review specificity, style, and behavior of engagement. To complement the PRAIB framework, we conduct a large-scale empirical study leveraging a dataset of 11,000 reviews generated by five proprietary and open-source models for 1,000 ICLR and NeurIPS papers. Spanning the 2021--2025 period, these machine-generated reviews are compared against original human feedback across diverse prompting strategies to identify systematic behavioral divergences. Our analysis reveals that the generated reviews diverge significantly from feedback provided by human reviewers: LLM ratings are less variable, positively biased, and overconfident, and their cross-reference patterns are model-dependent and distinct from human norms. Furthermore, when evaluated through PRAIB, we observe that LLMs tend to generate longer, more complex reviews, yet frequently overlook the atomic weaknesses noted by human reviewers. By characterizing where and how LLMs reviewing behavior departs from human norms, PRAIB provides the community with a diagnostic tool for identifying which aspects of the review process LLMs can reliably support today and which require further development before deployment.