# OmniNWM: Omniscient Driving Navigation World Models

arXiv: https://arxiv.org/abs/2510.18313 · HTML: https://mlsanity.com/paper/2510.18313/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 122 of its 142 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)
- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 30 of 122)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)

## Built on by (most cited first, 30 of 33)

- [DriveLaW:Unifying Planning and Video Generation in a Latent Driving World](https://mlsanity.com/paper/2512.23421/) (2025)
- [DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation](https://mlsanity.com/paper/2607.05147/) (2026)
- [Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving](https://mlsanity.com/paper/2603.24581/) (2026)
- [DVGT: Driving Visual Geometry Transformer](https://mlsanity.com/paper/2512.16919/) (2025)
- [DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving](https://mlsanity.com/paper/2603.24587/) (2026)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [Grounding World Simulation Models in a Real-World Metropolis](https://mlsanity.com/paper/2603.15583/) (2026)
- [ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.02714/) (2026)
- [Learning Vision-Language-Action World Models for Autonomous Driving](https://mlsanity.com/paper/2604.09059/) (2026)
- [UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation](https://mlsanity.com/paper/2607.05133/) (2026)
- [Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method](https://mlsanity.com/paper/2510.22973/) (2025)
- [RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework](https://mlsanity.com/paper/2604.15308/) (2026)
- [Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends](https://mlsanity.com/paper/2606.01164/) (2026)
- [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026)
- [FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving](https://mlsanity.com/paper/2603.14938/) (2026)
- [OpenWorldLib: A Unified Codebase and Definition of Advanced World Models](https://mlsanity.com/paper/2604.04707/) (2026)
- [PanoWorld: Geometry-Consistent Panoramic Video World Modeling](https://mlsanity.com/paper/2605.15391/) (2026)
- [ReWorld: Representation Learning for World Action Models](https://mlsanity.com/paper/2606.27504/) (2026)
- [SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations](https://mlsanity.com/paper/2604.07990/) (2026)
- [Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images](https://mlsanity.com/paper/2607.01633/) (2026)
- [Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation](https://mlsanity.com/paper/2607.06516/) (2026)
- [VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction](https://mlsanity.com/paper/2607.18078/) (2026)
- [Adaptive-WAM: Quality-Guided Early-Exit Planning from Intermediate Video-Diffusion Features](https://mlsanity.com/paper/2608.06008/) (2026)
- [DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding](https://mlsanity.com/paper/2603.19219/) (2026)
- [FDR-Occ: Factorized Dense Routing for Full-Spectrum 3D Occupancy Prediction](https://mlsanity.com/paper/2607.03822/) (2026)
- [SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction](https://mlsanity.com/paper/2607.04732/) (2026)
- [Visual Geometry Foundation-Aware Gaussians for Single-Frame Surround-View Driving Reconstruction](https://mlsanity.com/paper/2608.10682/) (2026)
- [PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesis](https://mlsanity.com/paper/2609.06948/) (2026)
- [HelloWorld: Towards Practical Applications of Generative Driving World Models](https://mlsanity.com/paper/2609.28931/) (2026)
- [CoDrive: Cross-Vehicle World-Consistent Video Generation with Precise Trajectory Control for Cooperative Driving](https://mlsanity.com/paper/2609.34749/) (2026)

## Abstract (arXiv)

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. However, existing methods are typically restricted to fragmented modality modeling, short-horizon drift, and imprecise action control, while lacking intrinsic mechanisms for policy evaluation. In this paper, we introduce OmniNWM, an Omniscient panoramic Navigation World Model that addresses all three dimensions within a consistent probabilistic framework. For State, OmniNWM generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy, ensuring pixel-level alignment across modalities with joint distribution modeling. To mitigate autoregressive exposure bias, we propose a structured panoramic forcing strategy to stabilize long-horizon generation via stochastic manifold thickening. For Action, we introduce canonical geometric action encoding with normalized panoramic Pl"ucker ray-maps. This representation decouples motion dynamics from sensor intrinsics, enabling precise, zero-shot trajectory control across heterogeneous datasets and camera configurations. For Reward, we derive intrinsic occupancy-grounded dense rewards directly from generated 3D volumes, establishing a reliable closed-loop simulation cycle for evaluating diverse planning agents. Extensive experiments demonstrate that OmniNWM achieves SOTA performance in generation fidelity and control precision, with remarkable zero-shot robustness to novel scenes on NuPlan and in-house datasets with distinct camera rigs. Project page is available at https://arlo0o.github.io/OmniNWM/.