mlsanityData to 7 Oct 2026

4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion

arXiv 2412.04462 ↗First posted 5 Dec 2024cs.CVcited by 27

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We propose 4Real-Video, a novel framework for generating 4D videos, organized as a grid of video frames with both time and viewpoint axes. In this grid, each row contains frames sharing the same timestep, while each column contains frames from the same viewpoint. We propose a novel two-stream architecture. One stream performs viewpoint updates on columns, and the other stream performs temporal updates on rows. After each diffusion transformer layer, a synchronization layer exchanges information between the two token streams. We propose two implementations of the synchronization layer, using either hard or soft synchronization. This feedforward architecture improves upon previous work in three ways: higher inference speed, enhanced visual quality (measured by FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency (measured by VideoScore and Dust3R-Confidence).

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 45 of its 57 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

45 of 57 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency24 Jul
  2. 2024Controlling Space and Time with Diffusion Models10 Jul
  3. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models11 Jun
  4. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generation31 May
  5. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistency28 Dec
  6. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models21 Dec
  7. 2023A Unified Approach for Text- and Image-guided 4D Scene Generation28 Nov
  8. 2023Animate124: Animating One Image to 4D Dynamic Scene24 Nov
  9. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  10. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  11. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
3 more
  1. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 30

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
18 more
  1. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  3. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  4. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  5. 2023Text-To-4D Dynamic Scene Generationcited by 204
  6. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  7. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  8. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  9. 2024Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairscited by 171
  10. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  11. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  12. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  13. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  14. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  15. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  16. 2024GIM: Learning Generalizable Image Matcher From Internet Videoscited by 72
  17. 2024DELTA: Dense Efficient Long-range 3D Tracking for any videocited by 62
  18. 2024GenXD: Generating Any 3D and 4D Scenescited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 10.

27 papers.

Sort:

By area:

  1. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 44
  2. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  3. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  4. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  5. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  6. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 20
  7. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusioncited by 15
  8. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  9. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  10. 2025Light-X: Generative 4D Video Rendering with Camera and Illumination Controlcited by 11
  11. 2025SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Timecited by 9
  12. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Modelscited by 7
15 more
  1. 2025IC-World: In-Context Generation for Shared World Modelingcited by 5
  2. 2026Full-4D: Generating Full-Scope 4D Scenes from a Single-View Videocited by 5
  3. 20254DSTR: Advancing Generative 4D Gaussians with Spatial-Temporal Rectification for High-Quality and Consistent 4D Generationcited by 4
  4. 2025OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesiscited by 4
  5. 2026BiMotion: B-spline Motion for Text-guided Dynamic 3D Character Generationcited by 4
  6. 2026Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformerscited by 3
  7. 2025T2Bs: Text-to-Character Blendshapes via Video Generationcited by 2
  8. 2025ChronosObserver: Taming 4D World with Hyperspace Diffusion Samplingcited by 2
  9. 2025SyncMV4D: Synchronized Multi-view Joint Diffusion of Appearance and Motion for Hand-Object Interaction Synthesiscited by 1
  10. 2026Ani3DHuman: Photorealistic 3D Human Animation with Self-guided Stochastic Samplingcited by 1
  11. 2026FlowLong: Inference-time Long Video Generation via Manifold-constrained Tweedie Matchingcited by 1
  12. 2026ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priorscited by 0
  13. 2026Orthogonal Spatial-temporal Distributional Transfer for 4D Generationcited by 0
  14. 2026VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matchingcited by 0
  15. 2026FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation
  2. 2026Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models
  3. 20254DVD: Cascaded Dense-view Video Diffusion Model for High-quality 4D Content Generation
  4. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
  5. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easy
  6. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
  7. 2025Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation
  8. 2026Beyond Pixels: From Video Priors to 4D Worlds