mlsanityData to 14 Oct 2026

SmartPlay: A Benchmark for LLMs as Intelligent Agents

arXiv 2310.01557 ↗First posted 2 Oct 2023cs.LG cs.AIcited by 125

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs' abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at github.com/Microsoft/SmartPlay

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 90 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

42 of 90 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 40

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
28 more
  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  6. 2023Large Language Modelscited by 780
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  10. 2023Textbooks Are All You Needcited by 569
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  13. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  14. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  15. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  16. 2023Language Models can Solve Computer Taskscited by 372
  17. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  18. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  19. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  20. 2020The NetHack Learning Environmentcited by 198
  21. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  22. 2020Imitating Interactive Intelligencecited by 55
  23. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  24. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  25. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 41
  26. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  27. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  28. 2021Language Models are Few-Shot Butlerscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 44; 2025: 53; 2026: 27.

125 papers.

Sort:

By area:

  1. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  2. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  3. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
  4. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  5. 2024Large Multimodal Agents: A Surveycited by 105
  6. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  7. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 95
  8. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 94
  9. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  10. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
  11. 2024Can large language models explore in-context?cited by 76
  12. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactionscited by 68
38 more
  1. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 67
  2. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  3. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaborationcited by 57
  4. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  5. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  6. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 46
  7. 2024Learning to Ask: When LLM Agents Meet Unclear Instructioncited by 42
  8. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 40
  9. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environmentscited by 36
  10. 2024VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraftcited by 35
  11. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 33
  12. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 31
  13. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 29
  14. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  15. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  16. 2024Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgmentscited by 26
  17. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
  18. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulationcited by 23
  19. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 23
  20. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  21. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  22. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  23. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  24. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  25. 2025Are Large Vision Language Models Good Game Players?cited by 19
  26. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 19
  27. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  28. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 15
  29. 2024AgentKit: Structured LLM Reasoning with Dynamic Graphscited by 13
  30. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  31. 2024Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputscited by 12
  32. 2024A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Modelscited by 12
  33. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 11
  34. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  35. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applicationscited by 10
  36. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  37. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  38. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 9

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  2. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  3. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  4. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors
  5. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
  6. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  7. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs
  8. 2025Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Games