mlsanityData to 14 Oct 2026

Video4DGen: Enhancing Video and 4D Generation through Mutual Optimization

arXiv 2504.04153 ↗First posted 5 Apr 2025cs.GR cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

The advancement of 4D (i.e., sequential 3D) generation opens up new possibilities for lifelike experiences in various applications, where users can explore dynamic objects or characters from any viewpoint. Meanwhile, video generative models are receiving particular attention given their ability to produce realistic and imaginative frames. These models are also observed to exhibit strong 3D consistency, indicating the potential to act as world simulators. In this work, we present Video4DGen, a novel framework that excels in generating 4D representations from single or multiple generated videos as well as generating 4D-guided videos. This framework is pivotal for creating high-fidelity virtual contents that maintain both spatial and temporal coherence. The 4D outputs generated by Video4DGen are represented using our proposed Dynamic Gaussian Surfels (DGS), which optimizes time-varying warping functions to transform Gaussian surfels (surface elements) from a static state to a dynamically warped state. We design warped-state geometric regularization and refinements on Gaussian surfels, to preserve the structural integrity and fine-grained appearance details. To perform 4D generation from multiple videos and capture representation across spatial, temporal, and pose dimensions, we design multi-video alignment, root pose optimization, and pose-guided frame sampling strategies. The leveraging of continuous warping fields also enables a precise depiction of pose, motion, and deformation over per-video frames. Further, to improve the overall fidelity from the observation of all camera poses, Video4DGen performs novel-view video generation guided by the 4D content, with the proposed confidence-filtered DGS to enhance the quality of generated sequences. With the ability of 4D and video generation, Video4DGen offers a powerful tool for applications in virtual reality, animation, and beyond.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 104 of its 120 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

104 of 120 references.

References · 104

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  9. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  10. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  11. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
38 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  2. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  3. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  4. 20242D Gaussian Splatting for Geometrically Accurate Radiance Fieldscited by 1,076
  5. 2020Neural Sparse Voxel Fieldscited by 959
  6. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
  7. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  8. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  9. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  10. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  11. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 746
  12. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  13. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  14. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  15. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  16. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  17. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  18. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  19. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  20. 2023SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Renderingcited by 616
  21. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 587
  22. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 511
  23. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 476
  24. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
  25. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 468
  26. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  27. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  28. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  29. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 405
  30. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  31. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 347
  32. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 346
  33. 2023Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesiscited by 344
  34. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  35. 2023Text-to-3D using Gaussian Splattingcited by 299
  36. 2021Mixture of Volumetric Primitives for Efficient Neural Renderingcited by 277
  37. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  38. 2020Single-View View Synthesis with Multiplane Imagescited by 263

50 most cited of 104 All 104 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  2. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Modelscited by 7
  3. 2026TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Imagecited by 2
  4. 2026Follow Your Track: Precise Skeleton Animation Controlled by 3D Trajectoriescited by 1
  5. 2026Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generationcited by 0