# CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection

arXiv: https://arxiv.org/abs/2509.04460 · HTML: https://mlsanity.com/paper/2509.04460/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Builds on (most cited first, 18 of 18)

- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [MGTBench: Benchmarking Machine-Generated Text Detection](https://mlsanity.com/paper/2303.14822/) (2023)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected?](https://mlsanity.com/paper/2401.05952/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [LLMDet: A Third Party Large Language Models Generated Text Detection Tool](https://mlsanity.com/paper/2305.15004/) (2023)
- [HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus](https://mlsanity.com/paper/2309.02731/) (2023)
- [Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review](https://mlsanity.com/paper/2502.19614/) (2025)
- [Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement](https://mlsanity.com/paper/2410.14259/) (2024)
- [OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution](https://mlsanity.com/paper/2504.11369/) (2025)
- ['Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews](https://mlsanity.com/paper/2410.09770/) (2024)

## Built on by (most cited first, 3 of 3)

- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection](https://mlsanity.com/paper/2607.17382/) (2026)
- [How Much Were You Told? Measuring External Information in Peer Reviews](https://mlsanity.com/paper/2609.28041/) (2026)

## Abstract (arXiv)

The growing integration of large language models (LLMs) into the peer review process presents potential risks to the fairness and reliability of scholarly evaluation. While LLMs offer valuable assistance for reviewers with language refinement, there is growing concern over their use to generate substantive review content. Existing general AI-generated text detectors are vulnerable to paraphrasing attacks and struggle to distinguish between surface language refinement and substantial content generation, suggesting that they primarily rely on stylistic cues. When applied to peer review, this limitation can result in unfairly suspecting reviews with permissible AI-assisted language enhancement, while failing to catch deceptively humanized AI-generated reviews. To address this, we propose a paradigm shift from style-based to content-based detection. Specifically, we introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews, covering six distinct modes of human-AI collaboration. Furthermore, we develop CoCoDet, an AI review detector via a multi-task learning framework, designed to achieve more accurate and robust detection of AI involvement in review content. Our work offers a practical foundation for evaluating the use of LLMs in peer review, and contributes to the development of more precise, equitable, and reliable detection methods for real-world scholarly applications. Our code and data will be publicly available at https://github.com/Y1hanChen/COCONUTS.