# A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challenges

arXiv: https://arxiv.org/abs/2403.10249 · HTML: https://mlsanity.com/paper/2403.10249/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 91 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 30 of 72)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models](https://mlsanity.com/paper/2311.05997/) (2023)
- [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)

## Built on by (most cited first, 22 of 22)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Odyssey: Empowering Minecraft Agents with Open-World Skills](https://mlsanity.com/paper/2407.15325/) (2024)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos](https://mlsanity.com/paper/2412.01800/) (2024)
- [Surveying the MLLM Landscape: A Meta-Review of Current Surveys](https://mlsanity.com/paper/2409.18991/) (2024)
- [On Large Language Models in National Security Applications](https://mlsanity.com/paper/2407.03453/) (2024)
- [Are Large Vision Language Models Good Game Players?](https://mlsanity.com/paper/2503.02358/) (2025)
- [StraGo: Harnessing Strategic Guidance for Prompt Optimization](https://mlsanity.com/paper/2410.08601/) (2024)
- [DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning](https://mlsanity.com/paper/2508.05405/) (2025)
- [VideoGameBunny: Towards vision assistants for video games](https://mlsanity.com/paper/2407.15295/) (2024)
- [ING-VP: MLLMs cannot Play Easy Vision-based Games Yet](https://mlsanity.com/paper/2410.06555/) (2024)
- [Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning](https://mlsanity.com/paper/2503.07002/) (2025)
- [Leveraging LLM Agents for Automated Video Game Testing](https://mlsanity.com/paper/2509.22170/) (2025)
- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Memory-Augmented State Machine Prompting: A Novel LLM Agent Framework for Real-Time Strategy Games](https://mlsanity.com/paper/2510.18395/) (2025)
- [Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities](https://mlsanity.com/paper/2510.19892/) (2025)
- [Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos](https://mlsanity.com/paper/2601.16471/) (2026)
- [Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset](https://mlsanity.com/paper/2412.20072/) (2024)
- [Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Play](https://mlsanity.com/paper/2604.17696/) (2026)

## Abstract (arXiv)

The swift evolution of Large-scale Models (LMs), either language-focused or multi-modal, has garnered extensive attention in both academy and industry. But despite the surge in interest in this rapidly evolving area, there are scarce systematic reviews on their capabilities and potential in distinct impactful scenarios. This paper endeavours to help bridge this gap, offering a thorough examination of the current landscape of LM usage in regards to complex game playing scenarios and the challenges still open. Here, we seek to systematically review the existing architectures of LM-based Agents (LMAs) for games and summarize their commonalities, challenges, and any other insights. Furthermore, we present our perspective on promising future research avenues for the advancement of LMs in games. We hope to assist researchers in gaining a clear understanding of the field and to generate more interest in this highly impactful research direction. A corresponding resource, continuously updated, can be found in our GitHub repository.