mlsanityData to 7 Oct 2026

NitroGen: An Open Foundation Model for Generalist Gaming Agents

arXiv 2601.02427 ↗First posted 4 Jan 2026cs.CV cs.AI cs.LGcited by 22

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

We introduce NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. We incorporate three key ingredients: 1) an internet-scale video-action dataset constructed by automatically extracting player actions from publicly available gameplay videos, 2) a multi-game benchmark environment that can measure cross-game generalization, and 3) a unified vision-action model trained with large-scale behavior cloning. NitroGen exhibits strong competence across diverse domains, including combat encounters in 3D action games, high-precision control in 2D platformers, and exploration in procedurally generated worlds. It transfers effectively to unseen games, achieving up to 52% relative improvement in task success rates over models trained from scratch. We release the dataset, evaluation suite, and model weights to advance research on generalist embodied agents.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 54 references that are in the map. First posted on arXiv 4 Jan 2026.

Builds on

42 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds12 Nov
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
27 more
  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  5. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  9. 2024Genie: Generative Interactive Environmentscited by 706
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  13. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  14. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  15. 2024Latent Action Pretraining from Videoscited by 302
  16. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  17. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  18. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  19. 2025Training Agents Inside of Scalable World Modelscited by 122
  20. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  21. 2024XFeat: Accelerated Features for Lightweight Image Matchingcited by 113
  22. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  23. 2018Imitating Latent Policies from Observationcited by 101
  24. 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 75
  25. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  26. 2024ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperationcited by 61
  27. 2024TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 22.

22 papers.

Sort:

By area:

  1. 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 12
  2. 2026AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Gamescited by 10
  3. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  4. 2026Prisma-World: Camera-Controllable Multi-Agent Video World Modelcited by 8
  5. 2026GameVerse: Can Vision-Language Models Learn from Video-based Reflection?cited by 7
  6. 2026Hallucination in World Models is Predictable and Preventablecited by 7
  7. 2026OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamicscited by 4
  8. 2026SPEAR: A Simulator for Photorealistic Embodied AI Researchcited by 3
  9. 2026WorldMind: Decoupled Game World Model for State-Aware NPC Behaviorcited by 3
  10. 2026Can Vision-Language Models Solve the Shell Game?cited by 2
  11. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 2
  12. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2
10 more
  1. 2026SPIKE: An Adaptive Dual Controller Framework for Cost-Efficient Long-Horizon Game Agentscited by 2
  2. 2026AI for Games in the Foundation Model Eracited by 2
  3. 2026Scalable Environments Drive Generalizable Agentscited by 1
  4. 2026Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraftcited by 1
  5. 2026GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agentscited by 0
  6. 2026Augmenting Game AI with Deep Reinforcement Learningcited by 0
  7. 2026DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Modelcited by 0
  8. 2026Cortex: Compact Behavior Cloning for Quake with Frozen Visual Featurescited by 0
  9. 2026PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Controlcited by 0
  10. 2026GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplaycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents
  2. 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents
  3. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
  4. 2026GameWAM: A World Action Model for Video Games
  5. 2019Towards Interactive Training of Non-Player Characters in Video Games
  6. 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models
  7. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
  8. 2026GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games