mlsanityData to 14 Oct 2026

AgentSims: An Open-Source Sandbox for Large Language Model Evaluation

arXiv 2308.04026 ↗First posted 8 Aug 2023cs.AIcited by 88

Not on the map. Why?

Cited by 66 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question. Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. We present AgentSims, an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in. Researchers can build their evaluation tasks by adding agents and buildings on an interactive GUI or deploy and test new support mechanisms, i.e. memory, planning and tool-use systems, by a few lines of codes. Our demo is available at https://agentsims.com .

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

17 of 24 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  8. 2023LIMA: Less Is More for Alignmentcited by 1,100
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  11. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
5 more
  1. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  2. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  3. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  4. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  5. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 26; 2025: 33; 2026: 17.

88 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  4. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  5. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  6. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  7. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  8. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  9. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  11. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  12. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
38 more
  1. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  2. 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 121
  3. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  4. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  5. 2023On Generative Agents in Recommendationcited by 90
  6. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  7. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  8. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  9. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 55
  10. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agentscited by 48
  11. 2024Enhance Reasoning for Large Language Models in the Game Werewolfcited by 46
  12. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  13. 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoningcited by 37
  14. 2025MCIP: Protecting MCP Safety via Model Contextual Integrity Protocolcited by 34
  15. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 33
  16. 2025Deep Research Bench: Evaluating AI Web Research Agentscited by 32
  17. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 31
  18. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  19. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  20. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  21. 2024AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaborationcited by 26
  22. 2023LARP: Language-Agent Role Play for Open-World Gamescited by 24
  23. 2024Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolfcited by 24
  24. 2024Synthesizing Post-Training Data for LLMs through Multi-Agent Simulationcited by 24
  25. 2024AgentLens: Visual Analysis for Agent Behaviors in LLM-based Autonomous Systemscited by 23
  26. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  27. 2025DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agentscited by 17
  28. 2023StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solvingcited by 13
  29. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROIcited by 12
  30. 2023Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulationscited by 11
  31. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  32. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  33. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  34. 2025Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Researchcited by 10
  35. 2025GATSim: Urban Mobility Simulation with Generative Agentscited by 10
  36. 2024Practical Considerations for Agentic LLM Systemscited by 9
  37. 2024Large Language Models as Minecraft Agentscited by 7
  38. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  2. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
  3. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  4. 2023AgentBench: Evaluating LLMs as Agents
  5. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
  6. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  7. 2026EvalBuilder: Can LLMs Build Efficient Benchmarks?
  8. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey