# Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review

arXiv: https://arxiv.org/abs/2502.19614 · HTML: https://mlsanity.com/paper/2502.19614/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 66 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 30 of 45)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors](https://mlsanity.com/paper/2405.07940/) (2024)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing](https://mlsanity.com/paper/2306.00622/) (2023)
- [TweepFake: about Detecting Deepfake Tweets](https://mlsanity.com/paper/2008.00036/) (2020)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Real or Fake? Learning to Discriminate Machine from Human Generated Text](https://mlsanity.com/paper/1906.03351/) (2019)
- [MGTBench: Benchmarking Machine-Generated Text Detection](https://mlsanity.com/paper/2303.14822/) (2023)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [A Design Space for Intelligent and Interactive Writing Assistants](https://mlsanity.com/paper/2403.14117/) (2024)

## Built on by (most cited first, 21 of 21)

- [SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?](https://mlsanity.com/paper/2605.30329/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [LLM-REVal: Can We Trust LLM Reviewers Yet?](https://mlsanity.com/paper/2510.12367/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Who's Your Judge? On the Detectability of LLM-Generated Judgments](https://mlsanity.com/paper/2509.25154/) (2025)
- [PeerPrism: Peer Evaluation Expertise vs Review-writing AI](https://mlsanity.com/paper/2604.14513/) (2026)
- [CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection](https://mlsanity.com/paper/2509.04460/) (2025)
- [APRES: An Agentic Paper Revision and Evaluation System](https://mlsanity.com/paper/2603.03142/) (2026)
- [LLMTrace: A Corpus for Classification and Fine-Grained Localization of AI-Written Text](https://mlsanity.com/paper/2509.21269/) (2025)
- [PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing](https://mlsanity.com/paper/2605.29815/) (2026)
- [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026)
- [BAID: A Benchmark for Bias Assessment of AI Detectors](https://mlsanity.com/paper/2512.11505/) (2025)
- [StackingNet: Collective Inference Across Independent AI Foundation Models](https://mlsanity.com/paper/2602.13792/) (2026)
- [Log-Likelihood, Simpson's Paradox, and the Detection of Machine-Generated Text](https://mlsanity.com/paper/2605.06294/) (2026)
- [Learning What Evaluators Value: A Reliable Approach to Modeling Evaluator Preferences](https://mlsanity.com/paper/2605.16615/) (2026)
- [Boosting Knowledge Graph Foundation Models via Enhanced Negative Sampling](https://mlsanity.com/paper/2605.27023/) (2026)
- [Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement](https://mlsanity.com/paper/2606.22166/) (2026)
- [Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains](https://mlsanity.com/paper/2607.03680/) (2026)
- [How Much Were You Told? Measuring External Information in Peer Reviews](https://mlsanity.com/paper/2609.28041/) (2026)
- [AI in Science: Early Insights](https://mlsanity.com/paper/2609.28504/) (2026)
- [Belief-Trajectory Energy: Measuring the Path to a Prediction](https://mlsanity.com/paper/2610.05114/) (2026)

## Abstract (arXiv)

Peer review is a critical process for ensuring the integrity of published scientific research. Confidence in this process is predicated on the assumption that experts in the relevant domain give careful consideration to the merits of manuscripts which are submitted for publication. With the recent rapid advancements in large language models (LLMs), a new risk to the peer review process is that negligent reviewers will rely on LLMs to perform the often time consuming process of reviewing a paper. However, there is a lack of existing resources for benchmarking the detectability of AI text in the domain of peer review. To address this deficiency, we introduce a comprehensive dataset containing a total of 788,984 AI-written peer reviews paired with corresponding human reviews, covering 8 years of papers submitted to each of two leading AI research conferences (ICLR and NeurIPS). We use this new resource to evaluate the ability of 18 existing AI text detection algorithms to distinguish between peer reviews fully written by humans and different state-of-the-art LLMs. Additionally, we explore a context-aware detection method called Anchor, which leverages manuscript content to detect AI-generated reviews, and analyze the sensitivity of detection models to LLM-assisted editing of human-written text. Our work reveals the difficulty of identifying AI-generated text at the individual peer review level, highlighting the urgent need for new tools and methods to detect this unethical use of generative AI. Our dataset is publicly available at: https://huggingface.co/datasets/IntelLabs/AI-Peer-Review-Detection-Benchmark.