mlsanityData to 7 Oct 2026

InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models

arXiv 2412.03934 ↗First posted 5 Dec 2024cs.CV cs.AI cs.GRcited by 53

On the map: World models → World models for autonomous driving

Abstract (arXiv)

We present InfiniCube, a scalable method for generating unbounded dynamic 3D driving scenes with high fidelity and controllability. Previous methods for scene generation either suffer from limited scales or lack geometric and appearance consistency along generated sequences. In contrast, we leverage the recent advancements in scalable 3D representation and video models to achieve large dynamic scene generation that allows flexible controls through HD maps, vehicle bounding boxes, and text descriptions. First, we construct a map-conditioned sparse-voxel-based 3D generative model to unleash its power for unbounded voxel world generation. Then, we re-purpose a video model and ground it on the voxel world through a set of carefully designed pixel-aligned guidance buffers, synthesizing a consistent appearance. Finally, we propose a fast feed-forward approach that employs both voxel and pixel branches to lift the dynamic videos to dynamic 3D Gaussians with controllable objects. Our method can generate controllable and realistic 3D driving scenes, and extensive experiments validate the effectiveness and superiority of our model.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 71 of its 83 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

71 of 83 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model14 Oct
  2. 2024OmniRe: Omni Urban Scene Reconstruction29 Aug
  3. 2024Dynamic 3D Gaussian Fields for Urban Areas5 Jun
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  5. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes23 May
  6. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  7. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep

Other references · 64

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
38 more
  1. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  3. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  6. 2024Depth Anything V2cited by 1,385
  7. 2022Planning-oriented Autonomous Drivingcited by 1,068
  8. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  9. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  10. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  11. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  12. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  13. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  14. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
  15. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  16. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  17. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  18. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
  19. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  20. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  21. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  22. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  23. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  24. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 244
  25. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  26. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  27. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  28. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  29. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  30. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  31. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  32. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  33. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 139
  34. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  35. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  36. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104
  37. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  38. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 31; 2026: 21.

53 papers.

Sort:

By area:

  1. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 162
  2. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 160
  3. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  4. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  5. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  6. 20253D and 4D World Modeling: A Surveycited by 64
  7. 2025A Survey of World Models for Autonomous Drivingcited by 60
  8. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  9. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  10. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  11. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  12. 2025Learning 3D Persistent Embodied World Modelscited by 29
38 more
  1. 2025ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulationcited by 28
  2. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  3. 2025Viser: Imperative, Web-based 3D Visualization in Pythoncited by 19
  4. 2025Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusioncited by 15
  5. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  6. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 14
  7. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14
  8. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  9. 2025ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidancecited by 11
  10. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  11. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  12. 2025GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generationcited by 10
  13. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  14. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  15. 2025Dreamland: Controllable World Creation with Simulator and Generative Modelscited by 5
  16. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  17. 2026ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Modelscited by 5
  18. 2025SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Modelcited by 3
  19. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  20. 2026MeshFlow: Mesh Generation with Equivariant Flow Matchingcited by 3
  21. 2025RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priorscited by 2
  22. 2025AutoScape: Geometry-Consistent Long-Horizon Scene Generationcited by 2
  23. 2025Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehiclescited by 2
  24. 2026WorldFlow3D: Flowing Through 3D Distributions for Unbounded World Generationcited by 2
  25. 2026HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenescited by 2
  26. 2026Sat2City v2: Native 3D City Asset Generation from a Single Satellite Imagecited by 2
  27. 2025NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfoldingcited by 1
  28. 2026ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generationcited by 1
  29. 2026OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligencecited by 1
  30. 2026FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Modelcited by 1
  31. 2026Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesiscited by 1
  32. 2026OpenLongTail: Generative Scaling of Long-Tail Driving Datacited by 1
  33. 2026VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoningcited by 0
  34. 2026OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Modelscited by 0
  35. 2026SEM-ROVER: Semantic Voxel-Guided Diffusion for Large-Scale Driving Scene Generationcited by 0
  36. 2026Map2World: Segment Map Conditioned Text to 3D World Generationcited by 0
  37. 2026AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyondcited by 0
  38. 2026InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Drivingcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  2. 2024DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation
  3. 20263DCarGen: Scalable 3D Car Generation via 3D-consistent Multi-view Synthesis
  4. 2026M4World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming
  5. 2026HelloWorld: Towards Practical Applications of Generative Driving World Models
  6. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  7. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control
  8. 2025SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model