mlsanityData to 7 Oct 2026

MindAgent: Emergent Gaming Interaction

arXiv 2309.09971 ↗First posted 18 Sep 2023cs.AI cs.HC cs.MAcited by 87

Not on the map. Why?

Cited by 52 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have the capacity of performing complex scheduling in a multi-agent system and can coordinate these agents into completing sophisticated tasks that require extensive collaboration. However, despite the introduction of numerous gaming frameworks, the community has insufficient benchmarks towards building general multi-agents collaboration infrastructure that encompass both LLM and human-NPCs collaborations. In this work, we propose a novel infrastructure - MindAgent - to evaluate planning and coordination emergent capabilities for gaming interaction. In particular, our infrastructure leverages existing gaming framework, to i) require understanding of the coordinator for a multi-agent system, ii) collaborate with human players via un-finetuned proper instructions, and iii) establish an in-context learning on few-shot prompt with feedback. Furthermore, we introduce CUISINEWORLD, a new gaming scenario and related benchmark that dispatch a multi-agent collaboration efficiency and supervise multiple agents playing the game simultaneously. We conduct comprehensive evaluations with new auto-metric CoS for calculating the collaboration efficiency. Finally, our infrastructure can be deployed into real-world gaming scenarios in a customized VR version of CUISINEWORLD and adapted in existing broader Minecraft gaming domain. We hope our findings on LLMs and the new infrastructure for general-purpose scheduling and coordination can help shed light on how such skills can be obtained by learning from large language corpora.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

40 of 59 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
28 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  7. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  8. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  10. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  11. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  12. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  13. 2023Large Language Models as General Pattern Machinescited by 196
  14. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  15. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
  16. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  17. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  18. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  19. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  20. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  21. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  22. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  23. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  24. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  25. 2023Putting ChatGPT's Medical Advice to the (Turing) Testcited by 23
  26. 2020Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Taskscited by 15
  27. 2023Towards Effective and Interpretable Human-Agent Collaboration in MOBA Games: A Communication Perspectivecited by 5
  28. 2023ArK: Augmented Reality with Knowledge Interactive Emergent Abilitycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 37; 2025: 23; 2026: 15.

87 papers.

Sort:

By area:

  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  2. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  4. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  5. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  6. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  7. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  8. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  9. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  10. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  11. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  12. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
38 more
  1. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  2. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  3. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4cited by 67
  4. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  5. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  7. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 46
  8. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  9. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  10. 2023GameGPT: Multi-agent Collaborative Framework for Game Developmentcited by 41
  11. 2024EduAgent: Generative Student Agents in Learningcited by 41
  12. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  13. 2024Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluationcited by 33
  14. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31
  15. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  16. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agentscited by 30
  17. 2024An Interactive Agent Foundation Modelcited by 25
  18. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  19. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  20. 2024S-Agents: Self-organizing Agents in Open-ended Environmentscited by 22
  21. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  22. 2025Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemmacited by 22
  23. 2024BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysiscited by 20
  24. 2024CityX: Controllable Procedural Content Generation for Unbounded 3D Citiescited by 20
  25. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  26. 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needscited by 16
  27. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16
  28. 2025Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMscited by 15
  29. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14
  30. 2024LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulationcited by 14
  31. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 13
  32. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  33. 2025WebGames: Challenging General-Purpose Web-Browsing AI Agentscited by 12
  34. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  35. 2026Code2Worlds: Empowering Coding LLMs for 4D World Generationcited by 12
  36. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 10
  37. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  38. 2023Vision-based Learning for Drones: A Surveycited by 9

50 most cited of 87 All 87 (JSON)