# Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds

arXiv: https://arxiv.org/abs/2511.08892 · HTML: https://mlsanity.com/paper/2511.08892/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- First posted on arXiv 12 Nov 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Built on by (most cited first, 21 of 21)

- [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](https://mlsanity.com/paper/2512.04797/) (2025)
- [GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents](https://mlsanity.com/paper/2604.07429/) (2026)
- [NitroGen: An Open Foundation Model for Generalist Gaming Agents](https://mlsanity.com/paper/2601.02427/) (2026)
- [Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playing](https://mlsanity.com/paper/2601.04575/) (2026)
- [GameVerse: Can Vision-Language Models Learn from Video-based Reflection?](https://mlsanity.com/paper/2603.06656/) (2026)
- [Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents](https://mlsanity.com/paper/2605.24598/) (2026)
- [Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks](https://mlsanity.com/paper/2602.01630/) (2026)
- [MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs](https://mlsanity.com/paper/2602.18600/) (2026)
- [Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning](https://mlsanity.com/paper/2605.00347/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [SPIKE: An Adaptive Dual Controller Framework for Cost-Efficient Long-Horizon Game Agents](https://mlsanity.com/paper/2605.18636/) (2026)
- [ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games](https://mlsanity.com/paper/2607.26712/) (2026)
- [IPR-1: Interactive Physical Reasoner](https://mlsanity.com/paper/2511.15407/) (2025)
- [MAIN-VLA: Modeling Abstraction of Intention and eNvironment for Vision-Language-Action Models](https://mlsanity.com/paper/2602.02212/) (2026)
- [SimpleTool: Parallel Decoding for Real-Time LLM Function Calling](https://mlsanity.com/paper/2603.00030/) (2026)
- [GameWAM: A World Action Model for Video Games](https://mlsanity.com/paper/2608.26200/) (2026)
- [UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City](https://mlsanity.com/paper/2608.27456/) (2026)
- [GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents](https://mlsanity.com/paper/2605.20246/) (2026)
- [GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay](https://mlsanity.com/paper/2609.25001/) (2026)
- [PUBG Ally: A Conversational Embodied Agent as an AI Teammate](https://mlsanity.com/paper/2609.29837/) (2026)
- [Player-1: Learning to Play from Synthetic Experience](https://mlsanity.com/paper/2610.20584/) (2026)

## Abstract (arXiv)

We introduce Lumine, the first open recipe for developing generalist agents capable of completing hours-long complex missions in real time within challenging 3D open-world environments. Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard-mouse actions and adaptively invokes reasoning only when necessary. Trained in Genshin Impact, Lumine successfully completes the entire five-hour Mondstadt main storyline on par with human-level efficiency and follows natural language instructions to perform a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. In addition to its in-domain performance, Lumine demonstrates strong zero-shot cross-game generalization. Without any fine-tuning, it accomplishes 100-minute missions in Wuthering Waves and the full five-hour first chapter of Honkai: Star Rail. These promising results highlight Lumine's effectiveness across distinct worlds and interaction dynamics, marking a concrete step toward generalist agents in open-ended environments.