mlsanityData to 14 Oct 2026

Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games

arXiv 2506.03610 ↗First posted 4 Jun 2025cs.AIcited by 31

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large Language Model (LLM) agents are reshaping the game industry, by enabling more intelligent and human-preferable characters. Yet, current game benchmarks fall short of practical needs: they lack evaluations of diverse LLM capabilities across various game genres, studies of agentic modules crucial for complex gameplay, and fine-tuning datasets to adapt pre-trained LLMs into gaming agents. To fill these gaps, we present Orak, a benchmark for training and evaluating LLM agents across 12 popular video games spanning all major genres. Using a plug-and-play interface built on Model Context Protocol (MCP), Orak supports systematic and reproducible studies of agentic modules in varied game scenarios. We further release a fine-tuning dataset of expert LLM gameplay trajectories covering multiple genres, turning general LLMs into effective game agents. Orak offers a united evaluation framework, including game leaderboards, LLM battle arenas, and fixablation studies of input modality, agentic strategies, and fine-tuning effects, establishing a foundation towards versatile gaming agents. Code and datasets are available at https://github.com/krafton-ai/Orak and https://huggingface.co/datasets/KRAFTON/Orak.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 46 of its 94 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

46 of 94 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
  2. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  3. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments8 Mar
  4. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb
  5. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  6. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents7 Jun
  7. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agents23 May
  8. 2024A Survey on Large Language Model-Based Game Agents2 Apr
  9. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments18 Mar
  10. 2024PokeLLMon: A Human-Parity Agent for Pokemon Battles with Large Language Models2 Feb
  11. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents19 Jan
  12. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agents2 Oct
1 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
21 more
  1. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 425
  2. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  4. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  5. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  6. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  7. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  8. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  9. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  10. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  11. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 100
  12. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 99
  13. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  14. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  15. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 57
  16. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
  17. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  18. 2023Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questionscited by 30
  19. 2025Are Large Vision Language Models Good Game Players?cited by 19
  20. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 11
  21. 2024Towards a Realistic Long-Term Benchmark for Open-Web Research Agentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 24.

31 papers.

Sort:

By area:

  1. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 77
  2. 2025Meta-RL Induces Exploration in Language Agentscited by 21
  3. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Taskscited by 15
  4. 2026Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playingcited by 11
  5. 2025Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chesscited by 9
  6. 2026GameVerse: Can Vision-Language Models Learn from Video-based Reflection?cited by 8
  7. 2026Workspace Optimization: How to Train Your Agentcited by 7
  8. 2026DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrationscited by 6
  9. 2026GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineerscited by 5
  10. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraftcited by 4
  11. 2025AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarioscited by 3
  12. 2025FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Gamescited by 3
19 more
  1. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 3
  2. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  3. 2026AI for Games in the Foundation Model Eracited by 3
  4. 2025Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interactioncited by 1
  5. 2025Can Agent Conquer Web? Exploring the Frontiers of ChatGPT Atlas Agent in Web Gamescited by 1
  6. 2026Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Timecited by 1
  7. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1
  8. 2026Can LLM Agents Infer World Models? Evidence from Agentic Automata Learningcited by 1
  9. 2026UrbanGround: From Local Perception to Spatial Agency in a Real-Scale Citycited by 1
  10. 2026RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agentscited by 0
  11. 2026Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Playcited by 0
  12. 2026PTCG-Bench: Can LLM Agents Master Pok'emon Trading Card Game?cited by 0
  13. 2026Experience-Sensitive Game Learning: A Behavioral Study of Humans and Language Agentscited by 0
  14. 2026Investigating Knowledge Transfer Across Interactive Dialogue Gamescited by 0
  15. 2026V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environmentscited by 0
  16. 2026CraftBench-UE: Deterministic Evaluation for Coding Agents in Unreal Enginecited by 0
  17. 2026GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplaycited by 0
  18. 2026FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Managementcited by 0
  19. 2026Separating Decision Time from Decision Quality in the Real-Time Gap of Distilled Deciders: Evidence from a Game and a Conveyor Simulatorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  2. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
  3. 2024Large Language Models as Agents in Two-Player Games
  4. 2026High-quality generation of dynamic game content via small language models: A proof of concept
  5. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
  6. 2026TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents
  7. 2025Leveraging LLM Agents for Automated Video Game Testing
  8. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments