# 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models

arXiv: https://arxiv.org/abs/2406.07472 · HTML: https://mlsanity.com/paper/2406.07472/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 65 of its 75 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 65)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)

## Built on by (most cited first, 30 of 64)

- [CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models](https://mlsanity.com/paper/2411.18613/) (2024)
- [VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control](https://mlsanity.com/paper/2407.12781/) (2024)
- [WorldScore: A Unified Evaluation Benchmark for World Generation](https://mlsanity.com/paper/2504.00983/) (2025)
- [AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers](https://mlsanity.com/paper/2411.18673/) (2024)
- [DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion](https://mlsanity.com/paper/2411.04928/) (2024)
- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning](https://mlsanity.com/paper/2411.05003/) (2024)
- [InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models](https://mlsanity.com/paper/2412.03934/) (2024)
- [SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generation](https://mlsanity.com/paper/2503.16396/) (2025)
- [Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation](https://mlsanity.com/paper/2509.19296/) (2025)
- [ResearchTown: Simulator of Human Research Community](https://mlsanity.com/paper/2412.17767/) (2024)
- [DreamDrive: Generative 4D Scene Modeling from Street View Images](https://mlsanity.com/paper/2501.00601/) (2024)
- [Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency](https://mlsanity.com/paper/2503.20785/) (2025)
- [4DNeX: Feed-Forward 4D Generative Modeling Made Easy](https://mlsanity.com/paper/2508.13154/) (2025)
- [4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion](https://mlsanity.com/paper/2412.04462/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance](https://mlsanity.com/paper/2505.21876/) (2025)
- [ShapeGen4D: Towards High Quality 4D Shape Generation from Videos](https://mlsanity.com/paper/2510.06208/) (2025)
- [Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesis](https://mlsanity.com/paper/2410.07155/) (2024)
- [Compositional Generative Model of Unbounded 4D Cities](https://mlsanity.com/paper/2501.08983/) (2025)
- [Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusion](https://mlsanity.com/paper/2503.22622/) (2025)
- [Simulating the Real World: A Unified Survey of Multimodal Generative Models](https://mlsanity.com/paper/2503.04641/) (2025)
- [AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories](https://mlsanity.com/paper/2602.14941/) (2026)
- [4Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation](https://mlsanity.com/paper/2506.18839/) (2025)
- [WideRange4D: Enabling High-Quality 4D Reconstruction with Wide-Range Movements and Scenes](https://mlsanity.com/paper/2503.13435/) (2025)
- [Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2411.19233/) (2024)
- [SS4D: Native 4D Generative Model via Structured Spacetime Latents](https://mlsanity.com/paper/2512.14284/) (2025)
- [Birth and Death of a Rose](https://mlsanity.com/paper/2412.05278/) (2024)
- [Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models](https://mlsanity.com/paper/2511.00503/) (2025)
- [RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation](https://mlsanity.com/paper/2607.06559/) (2026)

## Abstract (arXiv)

Existing dynamic scene generation methods mostly rely on distilling knowledge from pre-trained 3D generative models, which are typically fine-tuned on synthetic object datasets. As a result, the generated scenes are often object-centric and lack photorealism. To address these limitations, we introduce a novel pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. Our method begins by generating a reference video using the video generation model. We then learn the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video. To handle inconsistencies in the freeze-time video, we jointly learn a per-frame deformation to model these imperfections. We then learn the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video. The pipeline facilitates the generation of dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives, thereby setting a new standard in 4D scene generation.