mlsanityData to 7 Oct 2026

Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels

arXiv 2405.16822 ↗First posted 27 May 2024cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

Video generative models are receiving particular attention given their ability to generate realistic and imaginative frames. Besides, these models are also observed to exhibit strong 3D consistency, significantly enhancing their potential to act as world simulators. In this work, we present Vidu4D, a novel reconstruction model that excels in accurately reconstructing 4D (i.e., sequential 3D) representations from single generated videos, addressing challenges associated with non-rigidity and frame distortion. This capability is pivotal for creating high-fidelity virtual contents that maintain both spatial and temporal coherence. At the core of Vidu4D is our proposed Dynamic Gaussian Surfels (DGS) technique. DGS optimizes time-varying warping functions to transform Gaussian surfels (surface elements) from a static state to a dynamically warped state. This transformation enables a precise depiction of motion and deformation over time. To preserve the structural integrity of surface-aligned Gaussian surfels, we design the warped-state geometric regularization based on continuous warping fields for estimating normals. Additionally, we learn refinements on rotation and scaling parameters of Gaussian surfels, which greatly alleviates texture flickering during the warping process and enhances the capture of fine-grained appearance details. Vidu4D also contains a novel initialization state that provides a proper start for the warping fields in DGS. Equipping Vidu4D with an existing video generative model, the overall framework demonstrates high-fidelity text-to-4D generation in both appearance and geometry.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 86 of its 104 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

86 of 104 references.

References · 86

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  6. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  7. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  9. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  10. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  11. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  12. 20242D Gaussian Splatting for Geometrically Accurate Radiance Fieldscited by 1,061
38 more
  1. 2020Neural Sparse Voxel Fieldscited by 959
  2. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  4. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  5. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  6. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  7. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  8. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  9. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  10. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  11. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  12. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  13. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  14. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  15. 2023SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Renderingcited by 611
  16. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  17. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  18. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 470
  19. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  20. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  21. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  22. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  23. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  24. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 344
  25. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
  26. 2023Text-to-3D using Gaussian Splattingcited by 299
  27. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  28. 2021Mixture of Volumetric Primitives for Efficient Neural Renderingcited by 277
  29. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  30. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  31. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  32. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  33. 2021A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Posecited by 233
  34. 2020Free View Synthesiscited by 231
  35. 2022Tensor4D : Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Renderingcited by 217
  36. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  37. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  38. 2020AutoInt: Automatic Integration for Fast Neural Volume Renderingcited by 204

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 11.

38 papers.

Sort:

By area:

  1. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  3. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  4. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  5. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  6. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  7. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 33
  8. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 23
  9. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  10. 20244Dynamic: Text-to-4D Generation with Hybrid Priorscited by 15
  11. 2026ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusioncited by 15
  12. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
26 more
  1. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 13
  2. 2024Dynamic 2D Gaussians: Geometrically Accurate Radiance Fields for Dynamic Objectscited by 11
  3. 2026AdaGaR: Adaptive Gabor Representation for Dynamic Scene Reconstructioncited by 8
  4. 2024Birth and Death of a Rosecited by 7
  5. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  6. 2025HoliGS: Holistic Gaussian Splatting for Embodied View Synthesiscited by 7
  7. 2024Space-time 2D Gaussian Splatting for Accurate Surface Reconstruction under Complex Dynamic Scenescited by 6
  8. 20254DSloMo: 4D Reconstruction for High Speed Scene with Asynchronous Capturecited by 6
  9. 2024Adaptive and Temporally Consistent Gaussian Surfels for Multi-view Dynamic Reconstructioncited by 5
  10. 2025Gradient Variance Reveals Failure Modes in Flow-Based Generative Modelscited by 5
  11. 2026Choreographing a World of Dynamic Objectscited by 4
  12. 2026BiMotion: B-spline Motion for Text-guided Dynamic 3D Character Generationcited by 4
  13. 2024Towards a Science Exocortexcited by 3
  14. 2026SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generationcited by 3
  15. 2025T2Bs: Text-to-Character Blendshapes via Video Generationcited by 2
  16. 2025Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infillingcited by 2
  17. 2024Freeplane: Unlocking Free Lunch in Triplane-Based Sparse-View Reconstruction Modelscited by 1
  18. 2025Advances in 4D Generation: A Surveycited by 1
  19. 2025Restage4D: Reanimating Deformable 3D Reconstruction from a Single Videocited by 1
  20. 2026Fast 4D Mesh Generation by Spatio-Temporal Attention Chainscited by 1
  21. 2024Generative Diffusion Modeling: A Practical Handbookcited by 0
  22. 2025Generalized Dynamics Generation towards Scannable Physical World Modelcited by 0
  23. 2026Helix4D: Complex 4D Mesh Generationcited by 0
  24. 2026Feed-forward Motion In-betweening for Any 4Dcited by 0
  25. 2026Alignment Is All You Need For X-to-4D Generationcited by 0
  26. 2026Reconstructing the Dynamic World: A Representation-Centric View of 4D Scene Reconstructioncited by 0