# AgentReview: Exploring Peer Review Dynamics with LLM Agents

arXiv: https://arxiv.org/abs/2406.12708 · HTML: https://mlsanity.com/paper/2406.12708/

## Key facts

- Cited by 92 papers in the map (data through 7 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 39 of its 79 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 39)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations](https://mlsanity.com/paper/2307.02762/) (2023)
- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Can Large Language Model Agents Simulate Human Trust Behavior?](https://mlsanity.com/paper/2402.04559/) (2024)
- [LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting](https://mlsanity.com/paper/2402.16132/) (2024)
- [Dynamic Evaluation of Large Language Models by Meta Probing Agents](https://mlsanity.com/paper/2402.14865/) (2024)
- [Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment](https://mlsanity.com/paper/2109.09774/) (2021)
- [Image as Set of Points](https://mlsanity.com/paper/2303.01494/) (2023)
- [Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignment](https://mlsanity.com/paper/2311.09433/) (2023)
- [MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms](https://mlsanity.com/paper/2402.14154/) (2024)
- [Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach](https://mlsanity.com/paper/2211.06398/) (2022)
- [Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities](https://mlsanity.com/paper/2406.02787/) (2024)
- [Prototypical Reward Network for Data-Efficient RLHF](https://mlsanity.com/paper/2406.06606/) (2024)
- [PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners](https://mlsanity.com/paper/2310.02469/) (2023)
- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [Counterfactual Evaluation of Peer-Review Assignment Policies](https://mlsanity.com/paper/2305.17339/) (2023)
- [What makes a successful rebuttal in computer science conferences? : A perspective on social interaction](https://mlsanity.com/paper/2307.03371/) (2023)
- [A System-Level Analysis of Conference Peer Review](https://mlsanity.com/paper/2303.09020/) (2023)
- [Does double-blind peer-review reduce bias? Evidence from a top computer science conference](https://mlsanity.com/paper/2101.02701/) (2021)
- [Calibrating "Cheap Signals" in Peer Review without a Prior](https://mlsanity.com/paper/2312.07269/) (2023)
- [A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment](https://mlsanity.com/paper/2310.05995/) (2023)

## Built on by (most cited first, 30 of 92)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [AI-Researcher: Autonomous Scientific Innovation](https://mlsanity.com/paper/2505.18705/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research](https://mlsanity.com/paper/2505.19955/) (2025)
- [ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code](https://mlsanity.com/paper/2506.02314/) (2025)
- [AAAR-1.0: Assessing AI's Potential to Assist Research](https://mlsanity.com/paper/2410.22394/) (2024)
- [ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding](https://mlsanity.com/paper/2408.11363/) (2024)
- [Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions](https://mlsanity.com/paper/2505.07920/) (2025)
- [Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review](https://mlsanity.com/paper/2502.19614/) (2025)
- [AIGS: Generating Science from AI-Powered Automated Falsification](https://mlsanity.com/paper/2411.11910/) (2024)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Papers](https://mlsanity.com/paper/2509.19326/) (2025)
- [From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review](https://mlsanity.com/paper/2506.11343/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [am-ELO: A Stable Framework for Arena-based LLM Evaluation](https://mlsanity.com/paper/2505.03475/) (2025)
- [GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries](https://mlsanity.com/paper/2508.00033/) (2025)
- [LLM-REVal: Can We Trust LLM Reviewers Yet?](https://mlsanity.com/paper/2510.12367/) (2025)
- [AI Can Learn Scientific Taste](https://mlsanity.com/paper/2603.14473/) (2026)
- [Chinese Court Simulation with LLM-Based Agent System](https://mlsanity.com/paper/2508.17322/) (2025)
- [Simulating Dispute Mediation with LLM-Based Agents for Legal Research](https://mlsanity.com/paper/2509.06586/) (2025)
- [JoyAgent-JDGenie: Technical Report on the GAIA](https://mlsanity.com/paper/2510.00510/) (2025)
- [Who's Your Judge? On the Detectability of LLM-Generated Judgments](https://mlsanity.com/paper/2509.25154/) (2025)
- [DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning](https://mlsanity.com/paper/2601.07611/) (2026)
- [Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future](https://mlsanity.com/paper/2604.27924/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Can Agents Spontaneously Form a Society? Introducing a Novel Architecture for Generative Multi-Agents to Elicit Social Emergence](https://mlsanity.com/paper/2409.06750/) (2024)
- [Scaling Laws in Scientific Discovery with AI and Robot Scientists](https://mlsanity.com/paper/2503.22444/) (2025)
- [LLM-Agent-Controller: A Universal Multi-Agent Large Language Model System as a Control Engineer](https://mlsanity.com/paper/2505.19567/) (2025)

## Abstract (arXiv)

Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at https://github.com/Ahren09/AgentReview.