H3-World: Turning Language Understanding into World Control
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
We present H3-World, an efficient framework that turns the 33B MiniMax-H3 video generator into an interactive world model. Our key finding is that, as large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3, for example, already supports zero-shot control of character behavior and camera motion through natural-language instructions. Building on this, H3-World turns this coarse language interface into precise, temporally grounded world control, without introducing dedicated action modules. Specifically, we represent each action as a structured combination of character and camera instructions, and align them with the corresponding temporal video latents. To make the control temporally precise, we further introduce temporal attention routing, which restricts each instruction to its intended time interval and reduces control leakage across actions. Importantly, H3-World directly reuses the semantic representations learned during large-scale video pretraining and requires only lightweight adaptation. With only 8,000 gameplay samples, 10,000 LoRA optimization steps, and 0.199% trainable parameters, H3-World achieves effective character and camera control while preserving strong generation quality. It also generalizes to unseen scenarios. These results show that the control capabilities emerging in large video generators can be efficiently transformed into interactive world control.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 45 of its 50 references that are in the map. First posted on arXiv 1 Sep 2026.
Builds on
45 of 50 references.
Closest ancestors · 22
Same direction, newest first
- 2026WorldMind: Decoupled Game World Model for State-Aware NPC Behavior18 Aug
- 2026HelloWorld: Enabling Socially Interactive Characters in Video World Models5 Aug
- 2026StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation29 Jul
- 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU21 Jul
- 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memory16 Jun
- 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models22 May
- 2026Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models18 May
- 2026ReactiveGWM: Steering NPC in Reactive Game World Models14 May
- 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory10 Apr
- 2026Advancing Open-source World Models28 Jan
- 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling16 Dec
- 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model28 Nov
10 more
- 2025Matrix-game 2.0: An open-source real-time and streaming interactive world model18 Aug
- 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
- 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
- 2024GameGen-X: Interactive Open-world Game Video Generation1 Nov
- 2024Diffusion Models Are Real-Time Game Engines27 Aug
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
- 2024Genie: Generative Interactive Environments23 Feb
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
- 2023Learning Interactive Real-World Simulators9 Oct
- 2022Scalable Diffusion Models with Transformers19 Dec
Other references · 23
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2018World Modelscited by 1,513
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2020Mastering Atari with Discrete World Modelscited by 1,007
- 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
- 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
- 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
- 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
11 more
- 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
- 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
- 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
- 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
- 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
- 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 84
- 2025Yume-1.5: A Text-Controlled Interactive World Generation Modelcited by 84
- 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
- 2026Infinite Worlds with Versatile Interactionscited by 48
- 2026BadWorld: Adversarial Attacks on World Modelscited by 5
- 2026BadWAM: When World-Action Models Dream Right but Act Wrongcited by 4
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Controlcited by 4
- 2026AI for Games in the Foundation Model Eracited by 3
- 2026GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Modelscited by 1
- 2026LongLive-Plug: Once-for-All Distillation for Video Generationcited by 0
- 2026LongLive-Plug: Once-for-All Distillation for Video Generation29 Sep 2026
- 2026GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models22 Sep 2026
- 2026Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control15 Sep 2026
- 2026AI for Games in the Foundation Model Era15 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026Olaf-World: Orienting Latent Actions for Video World Modeling
- 2026WorldPlay2: Extending Real-Time Interactive World Models in Control and Horizon
- 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Models
- 2026Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer
- 2026World Action Models are Zero-shot Policies
- 2026SG-WAM: Text-Grounded and Spatial-aware Semantic Guidance for World-Action Models
- 2026DreamX-World 1.0: A General-Purpose Interactive World Model
- 2026World Tokens: Enhancing Embodied Policies with Training-Time World Modeling