mlsanityData to 14 Oct 2026

Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video

arXiv 2503.21761 ↗First posted 27 Mar 2025cs.CV cs.AI cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

This paper presents a unified approach to understanding dynamic scenes from casual videos. Large pretrained vision foundation models, such as vision-language, video depth prediction, motion tracking, and segmentation models, offer promising capabilities. However, training a single model for comprehensive 4D understanding remains challenging. We introduce Uni4D, a multi-stage optimization framework that harnesses multiple pretrained models to advance dynamic 3D modeling, including static/dynamic reconstruction, camera pose estimation, and dense 3D motion tracking. Our results show state-of-the-art performance in dynamic 4D modeling with superior visual quality. Notably, Uni4D requires no retraining or fine-tuning, highlighting the effectiveness of repurposing visual foundation models for 4D understanding.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

45 of 73 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Segment Anythingcited by 8,234
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  9. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  10. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
  11. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  12. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
33 more
  1. 2023CoTracker: It is Better to Track Togethercited by 522
  2. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  3. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  4. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  5. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
  6. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 382
  7. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 350
  8. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  9. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
  10. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 250
  11. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  12. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  13. 2024Global Structure-from-Motion Revisitedcited by 218
  14. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  15. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 197
  16. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  17. 2022Deep Patch Visual Odometrycited by 190
  18. 2020Robust Consistent Video Depth Estimationcited by 188
  19. 2019ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residualscited by 181
  20. 2024MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffoldscited by 179
  21. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 173
  22. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
  23. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 124
  24. 2022ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wildcited by 92
  25. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 91
  26. 2023Dense Optical Tracking: Connecting the Dotscited by 61
  27. 2021Weakly Supervised Learning of Rigid 3D Scene Flowcited by 60
  28. 2024LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometrycited by 56
  29. 2019Deep Rigid Instance Scene Flowcited by 54
  30. 2022CASA: Category-agnostic Skeletal Animal Reconstructioncited by 35
  31. 2023Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesiscited by 20
  32. 2023Temporally Consistent Online Depth Estimation Using Point-Based Fusioncited by 12
  33. 2023SLoMo: A General System for Legged Robot Motion Imitation from Casual Videoscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 15.

29 papers.

Sort:

By area:

  1. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 67
  2. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 35
  3. 2025VGGT4D: Mining Motion Cues in Visual Geometry Transformers for 4D Scene Reconstructioncited by 28
  4. 2026VGGT-Omegacited by 26
  5. 2025Viser: Imperative, Web-based 3D Visualization in Pythoncited by 23
  6. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  7. 2026Vista4D: Video Reshooting with 4D Point Cloudscited by 18
  8. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 14
  9. 2026Track4World: Feedforward World-centric Dense 3D Tracking of All Pixelscited by 13
  10. 2026MoRe: Motion-aware Feed-forward 4D Reconstruction Transformercited by 9
  11. 2025PAGE-4D: Disentangled pose and geometry estimation for vggt-4d perceptioncited by 8
  12. 2025E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Modelscited by 7
17 more
  1. 2025TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixelscited by 7
  2. 2025DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modelingcited by 5
  3. 2025HuPrior3R: Incorporating Human Priors for Better 3D Dynamic Reconstruction from Monocular Videoscited by 3
  4. 2026DAGE: Dual-Stream Architecture for Efficient and Fine-Grained Geometry Estimationcited by 3
  5. 2026FreeArtGS: Articulated Gaussian Splatting Under Free-moving Scenariocited by 3
  6. 2025Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generationcited by 2
  7. 2026MoRight: Motion Control Done Rightcited by 2
  8. 2025Visual Sync: Multi-Camera Synchronization via Cross-View Object Motioncited by 1
  9. 2026AeroDGS: Physically Consistent Dynamic Gaussian Splatting for Single-Sequence Aerial 4D Reconstructioncited by 1
  10. 2026Preserve, Reveal, Expand: Faithful 4D Video Editing with Region-Aware Conditioningcited by 1
  11. 2026UniQuery4R: Unified 4D Scene Reconstruction from a Single Querycited by 1
  12. 2025InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertioncited by 0
  13. 2026ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priorscited by 0
  14. 2026Controllable Video Object Insertion via Multi-View Priorscited by 0
  15. 20264DVGGT-D: 4D Visual Geometry Transformer with Improved Dynamic Depth Estimationcited by 0
  16. 2026MoSE3: Learning World-Space SE(3) at Every Pixelcited by 0
  17. 20264LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modelingcited by 0