# Zero-shot reasoning for simulating scholarly peer-review

arXiv: https://arxiv.org/abs/2510.02027 · HTML: https://mlsanity.com/paper/2510.02027/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 96 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models](https://mlsanity.com/paper/2212.10846/) (2022)
- [The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning](https://mlsanity.com/paper/2305.14045/) (2023)
- [Parsing Argumentation Structures in Persuasive Essays](https://mlsanity.com/paper/1604.07370/) (2016)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation](https://mlsanity.com/paper/2303.03480/) (2023)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning](https://mlsanity.com/paper/2404.08680/) (2024)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [SiReRAG: Indexing Similar and Related Information for Multihop Reasoning](https://mlsanity.com/paper/2412.06206/) (2024)
- [Can ChatGPT be used to generate scientific hypotheses?](https://mlsanity.com/paper/2304.12208/) (2023)
- [Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts](https://mlsanity.com/paper/2311.09050/) (2023)
- [Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision](https://mlsanity.com/paper/2406.00197/) (2024)
- [MALADE: Orchestration of LLM-powered Agents with Retrieval Augmented Generation for Pharmacovigilance](https://mlsanity.com/paper/2408.01869/) (2024)
- [Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions](https://mlsanity.com/paper/2410.02028/) (2024)
- [Evaluating the Predictive Capacity of ChatGPT for Academic Peer Review Outcomes Across Multiple Platforms](https://mlsanity.com/paper/2411.09763/) (2024)
- [Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models](https://mlsanity.com/paper/2502.14904/) (2025)

## Abstract (arXiv)

Scholarly publishing requires scalable scrutiny supported by auditable evidence. This paper presents a two-component benchmark of xPeer, the peer-review simulation engine delivered through the xPeerd.com web front-end. The operational component analyzes 352 of 500 simulation records retained under stable-task criteria across disciplines and review modes. The human-reference component releases 1,108 version-1 F1000Research manuscript records with linked human reports and applies a prespecified two-human/two-xPeer comparison. Human-review text and recommendations remained outside the generation input, and source joining occurred after xPeer outputs had been persisted. This procedure defines workflow-level review withholding; prior model exposure falls outside the recorded design. Among 802 records with exactly two human reports, 271 contained two usable xPeer reviewer fields, giving a complete-pair availability rate of 33.8%. Under deterministic extraction rules, median manuscript-level report length was 1,889 words for xPeer and 763 for humans, while median concern count was 41 and 13, respectively. xPeer reports showed higher targeting, category coverage, and executability. Human reports showed higher explicit-reasoning language, lexical manuscript attestation, taxonomy-based scientific relevance, and lower mean within-source redundancy. Cross-source lexical concern matching and recommendation agreement were low. The evidence therefore defines distinct observable review profiles and a transparent reproducibility baseline. Scientific correctness of individual concerns, autonomous editorial use, and cross-system superiority require expert adjudication and common-protocol testing. The study-level dataset and exact version-pinned reproducibility record are archived on Zenodo