mlsanityData to 14 Oct 2026

Generative Adversarial Reviews: When LLMs Become the Critic

arXiv 2412.10415 ↗First posted 9 Dec 2024cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The peer review process is fundamental to scientific progress, determining which papers meet the quality standards for publication. Yet, the rapid growth of scholarly production and increasing specialization in knowledge areas strain traditional scientific feedback mechanisms. In light of this, we introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers. To enable generative reviewers, we design an architecture that extends a large language model with memory capabilities and equips agents with reviewer personas derived from historical data. Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information - linking ideas with evidence and technical details. GAR's review process leverages external knowledge to evaluate paper novelty, followed by detailed assessment using the graph representation and multi-round assessment. Finally, a meta-reviewer aggregates individual reviews to predict the acceptance decision. Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes. Beyond mere performance comparison, we conduct insightful experiments, such as evaluating the impact of reviewer expertise and examining fairness in reviews. By offering early expert-level feedback, typically restricted to a limited group of researchers, GAR democratizes access to transparent and in-depth evaluation.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 37 of its 67 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

37 of 67 references.

References · 37

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  6. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  7. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  10. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 520
  11. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  12. 2018From Louvain to Leiden: guaranteeing well-connected communitiescited by 385
25 more
  1. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  2. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  3. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 233
  4. 2023Playing repeated games with Large Language Modelscited by 217
  5. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  6. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  7. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  8. 2021The Values Encoded in Machine Learning Researchcited by 172
  9. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 166
  10. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  11. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  12. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  13. 2023AnglE-optimized Text Embeddingscited by 99
  14. 2024Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent Systemcited by 68
  15. 2023Simulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithmscited by 65
  16. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 46
  17. 2024AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviewscited by 44
  18. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
  19. 2022A Decade of Knowledge Graphs in Natural Language Processing: A Surveycited by 40
  20. 2023GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Studycited by 40
  21. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  22. 2023Large Language Models for Scientific Synthesis, Inference and Explanationcited by 39
  23. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  24. 2022REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Researchcited by 20
  25. 2024Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Reviewcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  2. 2025Large language models for automated scholarly paper review: A surveycited by 61
  3. 2025CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulationcited by 28
  4. 2025Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviewscited by 24
  5. 2025Literature-Grounded Novelty Assessment of Scientific Ideascited by 19
  6. 2026Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideascited by 10
  7. 2025MobileCity: An Efficient Framework for Large-Scale Urban Behavior Simulationcited by 5
  8. 2026DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttalcited by 5
  9. 2026When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysiscited by 2
  10. 2026Beyond Detection: Governing GenAI in Academic Peer Review as a Sociotechnical Challengecited by 0
  11. 2026CityReal: Human-Aligned Urban Behavior and City Dynamics Simulation with Large-Scale LLM Agentscited by 0
  12. 2026NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessmentcited by 0