# Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026

arXiv: https://arxiv.org/abs/2609.19420 · HTML: https://mlsanity.com/paper/2609.19420/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2026.

## Built on by (most cited first, 2 of 2)

- [Verify Less, Evolve More: Training Idea-Level Critics for Verification-Efficient ML Evolving Agents](https://mlsanity.com/paper/2610.08993/) (2026)
- [How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting](https://mlsanity.com/paper/2610.18435/) (2026)

## Abstract (arXiv)

LLMs are rapidly reshaping peer review, making it important to understand how reviewers use them in practice and how different LLM-use policies affect review outcomes. We investigate these questions through a randomized experiment and an anonymous post-survey at ICML 2026, a major machine learning conference involving over 24,000 papers and 17,000 reviewers. Reviewers were assigned to either a conservative policy prohibiting all LLM use or a permissive policy allowing limited assistance, with randomization among a subset of main-track papers and reviewers. Policy assignment had near-zero effects on final paper decisions, paper scores, and reviewer confidence, although reviews under the permissive policy were 5.5-7% longer. Post-survey responses (N=1,486) revealed diverse attitudes toward LLMs and substantial noncompliance: 22.5% of conservative-policy reviewers reported using an LLM despite the prohibition, and 36.5% of permissive-policy reviewers reported at least one explicitly disallowed use. We discuss implications for future peer-review policy and tool design.