mlsanityData to 7 Oct 2026

Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents

arXiv 2510.23691 ↗First posted 27 Oct 2025cs.AIcited by 23

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

We present Game-TARS, a generalist game agent trained with a unified, scalable action space anchored to human-aligned native keyboard-mouse inputs. Unlike API- or GUI-based approaches, this paradigm enables large-scale continual pre-training across heterogeneous domains, including OS, web, and simulation games. Game-TARS is pre-trained on over 500B tokens with diverse trajectories and multimodal data. Key techniques include a decaying continual loss to reduce causal confusion and an efficient Sparse-Thinking strategy that balances reasoning depth and inference cost. Experiments show that Game-TARS achieves about 2 times the success rate over the previous sota model on open-world Minecraft tasks, is close to the generality of fresh humans in unseen web 3d games, and outperforms GPT-5, Gemini-2.5-Pro, and Claude-4-Sonnet in FPS benchmarks. Scaling results on training-time and test-time confirm that the unified action space sustains improvements when scaled to cross-game and multimodal data. Our results demonstrate that simple, scalable action representations combined with large-scale pre-training provide a promising path toward generalist agents with broad computer-use abilities.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 59 of its 76 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

59 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft13 Sep
  2. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently21 Apr
  3. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  4. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks7 Aug
  5. 2024OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents27 Jun
  6. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  7. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models10 Nov
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 50

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  11. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  12. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
38 more
  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2022A Generalist Agentcited by 802
  7. 2024Genie: Generative Interactive Environmentscited by 706
  8. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  9. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  10. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  13. 2025Seed1.5-VL Technical Reportcited by 367
  14. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  15. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  16. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  17. 2019Causal Confusion in Imitation Learningcited by 286
  18. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  19. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  20. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  21. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  22. 2024RT-H: Action Hierarchies Using Languagecited by 204
  23. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  24. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  25. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  26. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  27. 2025GR-3 Technical Reportcited by 110
  28. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  29. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  30. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  31. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  32. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  33. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  34. 2018ViZDoom Competitions: Playing Doom from Pixelscited by 50
  35. 2023Skill Reinforcement Learning and Planning for Open-World Long-Horizon Taskscited by 35
  36. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 29
  37. 2025JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mousecited by 27
  38. 2025VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoocited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 20.

23 papers.

Sort:

By area:

  1. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  2. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 23
  3. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  4. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  5. 2026Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playingcited by 10
  6. 2026AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Gamescited by 10
  7. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  8. 2026GameVerse: Can Vision-Language Models Learn from Video-based Reflection?cited by 7
  9. 2026OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamicscited by 4
  10. 2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
  11. 2025IPR-1: Interactive Physical Reasonercited by 2
  12. 2026MAIN-VLA: Modeling Abstraction of Intention and eNvironment for Vision-Language-Action Modelscited by 2
11 more
  1. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 2
  2. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
  3. 2026AI for Games in the Foundation Model Eracited by 2
  4. 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigationcited by 1
  5. 2026ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Gamescited by 1
  6. 2025Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learningcited by 0
  7. 2026Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Playcited by 0
  8. 2026GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agentscited by 0
  9. 2026GameWAM: A World Action Model for Video Gamescited by 0
  10. 2026GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplaycited by 0
  11. 2026VISTA: A Visual Harness for Reasoning in an Interactive Worldcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
  2. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
  3. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agents
  4. 2025Training Agents Inside of Scalable World Models
  5. 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
  6. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft
  7. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  8. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games