# More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review

arXiv: https://arxiv.org/abs/2609.05788 · HTML: https://mlsanity.com/paper/2609.05788/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 11 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://mlsanity.com/paper/2412.11948/) (2024)
- [TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review](https://mlsanity.com/paper/2506.07642/) (2025)

## Abstract (arXiv)

Peer-review feedback often arrives too late for authors to make meaningful revisions. We study an author-facing LLM system that moves part of this stress test before submission: it generates a broad pool of atomic concerns and compresses them into a short report. We evaluate agreement with historical reviews and, separately, the possible validity of concerns they omit. From 10,000 ICLR 2026 submissions, we use 3,398 manuscripts with accessible versions that predate review. On a ten-paper diagnostic, independent sampling covers 44.9% of historical issues; deduplication and refill reaches 78.7% strict and 84.9% seriousness-weighted coverage, at 3.6× more requests and 5.2× more tokens. A hidden Top-32 Oracle preserves the full 79.3% weighted coverage of a 256-candidate pool, but paper-only selectors retain only 40--44%. LLM review therefore provides broad coverage with a large candidate pool but compresses poorly; ablations identify representative selection and matcher sensitivity as the main sources of this gap.