mlsanityData to 7 Oct 2026

AgentReview: Exploring Peer Review Dynamics with LLM Agents

arXiv 2406.12708 ↗First posted 18 Jun 2024cs.CLcited by 92

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms. Our code is available at https://github.com/Ahren09/AgentReview.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 79 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

39 of 79 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  6. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  7. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 195
  8. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  9. 2024Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviewscited by 160
  10. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  11. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  12. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
27 more
  1. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  2. 2024LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Promptingcited by 56
  3. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  4. 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 49
  5. 2023Image as Set of Pointscited by 43
  6. 2023Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignmentcited by 27
  7. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  8. 2022Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approachcited by 23
  9. 2024Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilitiescited by 22
  10. 2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
  11. 2023PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learnerscited by 13
  12. 2023Semi-Offline Reinforcement Learning for Optimized Text Generationcited by 12
  13. 2023Counterfactual Evaluation of Peer-Review Assignment Policiescited by 11
  14. 2023What makes a successful rebuttal in computer science conferences? : A perspective on social interactioncited by 10
  15. 2023A System-Level Analysis of Conference Peer Reviewcited by 9
  16. 2021Does double-blind peer-review reduce bias? Evidence from a top computer science conferencecited by 8
  17. 2023Calibrating "Cheap Signals" in Peer Review without a Priorcited by 8
  18. 2023A One-Size-Fits-All Approach to Improving Randomness in Paper Assignmentcited by 6
  19. 2023Surveying (Dis)Parities and Concerns of Compute Hungry NLP Researchcited by 5
  20. 2023Peer Prediction for Peer Review: Designing a Marketplace for Ideascited by 4
  21. 2023Reviewer assignment problem: A scoping reviewcited by 4
  22. 2023Estimating the Causal Effect of Early ArXiving on Paper Acceptancecited by 4
  23. 2023Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer Reviewcited by 3
  24. 2024Distinguishing articles in questionable and non-questionable journals using quantitative indicators associated with qualitycited by 2
  25. 2024A FAIR and Free Prompt-based Research Assistantcited by 2
  26. 2021Do conspicuous manuscripts experience shorter time in the duration of peer review?cited by 1
  27. 2023Safeguarding Scientific Integrity: Examining Conflicts of Interest in the Peer Review Processcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 41; 2026: 43.

92 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 97
  3. 2025AI-Researcher: Autonomous Scientific Innovationcited by 88
  4. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  5. 2025Large language models for automated scholarly paper review: A surveycited by 59
  6. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  7. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  8. 2024AAAR-1.0: Assessing AI's Potential to Assist Researchcited by 24
  9. 2024ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understandingcited by 23
  10. 2025Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussionscited by 19
  11. 2025Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Reviewcited by 18
  12. 2024AIGS: Generating Science from AI-Powered Automated Falsificationcited by 17
38 more
  1. 2025How Far Are AI Scientists from Changing the World?cited by 15
  2. 2025Unveiling the Merits and Defects of LLMs in Automatic Review Generation for Scientific Paperscited by 12
  3. 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Reviewcited by 11
  4. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  5. 2025am-ELO: A Stable Framework for Arena-based LLM Evaluationcited by 8
  6. 2025GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Librariescited by 8
  7. 2025LLM-REVal: Can We Trust LLM Reviewers Yet?cited by 8
  8. 2026AI Can Learn Scientific Tastecited by 8
  9. 2025Chinese Court Simulation with LLM-Based Agent Systemcited by 7
  10. 2025Simulating Dispute Mediation with LLM-Based Agents for Legal Researchcited by 7
  11. 2025JoyAgent-JDGenie: Technical Report on the GAIAcited by 7
  12. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 6
  13. 2026DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoningcited by 6
  14. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  15. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  16. 2024Can Agents Spontaneously Form a Society? Introducing a Novel Architecture for Generative Multi-Agents to Elicit Social Emergencecited by 5
  17. 2025Scaling Laws in Scientific Discovery with AI and Robot Scientistscited by 5
  18. 2025LLM-Agent-Controller: A Universal Multi-Agent Large Language Model System as a Control Engineercited by 5
  19. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  20. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  21. 2025To Mask or to Mirror: Human-AI Alignment in Collective Reasoningcited by 5
  22. 2026Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performancecited by 5
  23. 2026RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generationcited by 5
  24. 2025Can AI Validate Science? Benchmarking LLMs for Accurate Scientific Claim rightarrow Evidence Reasoningcited by 4
  25. 2025AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environmentscited by 4
  26. 2025Preacher: Paper-to-Video Agentic Systemcited by 4
  27. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  28. 2026RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mindcited by 4
  29. 2024Empowering Interdisciplinary Insights with Dynamic Graph Embedding Trajectoriescited by 3
  30. 2025Deliberate Planning in Language Models with Symbolic Representationcited by 3
  31. 2025FML-bench: Benchmarking Machine Learning Agents for Scientific Researchcited by 3
  32. 2025ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentationcited by 3
  33. 2026Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviewscited by 3
  34. 2025Bridging Social Psychology and LLM Reasoning: Conflict-Aware Meta-Review Generation via Cognitive Alignmentcited by 2
  35. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  36. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  37. 2025Mirroring Users: Towards Building Preference-aligned User Simulator with User Feedback in Recommendationcited by 2
  38. 2025Multi-Agent LLMs for Generating Research Limitationscited by 2

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Passive Generation to Investigation: A Proactive Scientific Peer Review Agent
  2. 2025AI and the Future of Academic Peer Review
  3. 2026Benchmarking Agentic Review Systems
  4. 2026DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review
  5. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation
  6. 2025LiRA: A Multi-Agent Framework for Reliable and Readable Literature Review Generation
  7. 2025BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
  8. 2026Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results