RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
World foundation models aim to simulate the evolution of the real world with physically plausible behavior. Unlike prior methods that handle spatial and temporal correlations separately, we propose RAYNOVA, a geometry-agonistic multiview world model for driving scenarios that employs a dual-causal autoregressive framework. It follows both scale-wise and temporal topological orders in the autoregressive process, and leverages global attention for unified 4D spatio-temporal reasoning. Different from existing works that impose strong 3D geometric priors, RAYNOVA constructs an isotropic spatio-temporal representation across views, frames, and scales based on relative Pl"ucker-ray positional encoding, enabling robust generalization to diverse camera setups and ego motions. We further introduce a recurrent training paradigm to alleviate distribution drift in long-horizon video generation. RAYNOVA achieves state-of-the-art multi-view video generation results on nuScenes, while offering higher throughput and strong controllability under diverse input conditions, generalizing to novel views and camera configurations without explicit 3D scene representation. Our code will be released at https://raynova-ai.github.io/.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 66 of its 78 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
66 of 78 references.
Closest ancestors · 6
Same direction, newest first
- 2024WorldSimBench: Towards Video Generation Models as World Simulators23 Oct
- 2024Genie: Generative Interactive Environments23 Feb
- 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation6 Feb
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
- 2022MAGVIT: Masked Generative Video Transformer10 Dec
- 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
Other references · 60
- 2017Attention Is All You Needcited by 59,932
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2018World Modelscited by 1,444
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
38 more
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
- 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
- 2022Autoregressive Image Generation using Residual Quantizationcited by 706
- 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
- 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
- 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
- 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
- 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
- 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
- 2022Transformers are Sample-Efficient World Modelscited by 317
- 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
- 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
- 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
- 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
- 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
- 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
- 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
- 2024Rotary Position Embedding for Vision Transformercited by 226
- 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
- 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 218
- 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
- 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
- 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
- 2024OmniRe: Omni Urban Scene Reconstructioncited by 159
- 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
- 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
- 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
- 2022Object Scene Representation Transformercited by 90
- 2023SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detectioncited by 83
- 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
- 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
- 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
- 2023ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modelingcited by 60
- 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
- 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026URoPE: Universal Relative Position Embedding across Geometric Spacescited by 5
- 2026Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolutioncited by 4
- 2026SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformerscited by 3
- 2026MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generationcited by 1
- 2026MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation1 Sep 2026
- 2026SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers25 Jun 2026
- 2026Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution25 May 2026
- 2026URoPE: Universal Relative Position Embedding across Geometric Spaces20 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2025Video World Models with Long-term Spatial Memory
- 2025SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world models
- 2026AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video
- 2025DeepVerse: 4D Autoregressive Video Generation as a World Model
- 2026VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model
- 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling
- 2025LongVie 2: Multimodal Controllable Ultra-Long Video World Model
- 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modeling