mlsanityData to 7 Oct 2026

MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds

arXiv 2405.17421 ↗First posted 27 May 2024cs.CV cs.GRcited by 177

Not on the map. Why?

Abstract (arXiv)

We introduce 4D Motion Scaffolds (MoSca), a modern 4D reconstruction system designed to reconstruct and synthesize novel views of dynamic scenes from monocular videos captured casually in the wild. To address such a challenging and ill-posed inverse problem, we leverage prior knowledge from foundational vision models and lift the video data to a novel Motion Scaffold (MoSca) representation, which compactly and smoothly encodes the underlying motions/deformations. The scene geometry and appearance are then disentangled from the deformation field and are encoded by globally fusing the Gaussians anchored onto the MoSca and optimized via Gaussian Splatting. Additionally, camera focal length and poses can be solved using bundle adjustment without the need of any other pose estimation tools. Experiments demonstrate state-of-the-art performance on dynamic rendering benchmarks and its effectiveness on real videos.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 92 of its 127 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

92 of 127 references.

References · 92

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  11. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  12. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
38 more
  1. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  2. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  3. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  4. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  5. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  7. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  8. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  9. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  10. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  11. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  12. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  13. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  14. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  15. 2023CoTracker: It is Better to Track Togethercited by 516
  16. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  17. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 470
  18. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 470
  19. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  20. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  21. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  22. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  23. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 375
  24. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 344
  25. 2023Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesiscited by 341
  26. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
  27. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  28. 2024Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenescited by 226
  29. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  30. 2021HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fieldscited by 210
  31. 2023Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particlecited by 210
  32. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
  33. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  34. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  35. 20233DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splattingcited by 195
  36. 2023Robust Dynamic Radiance Fieldscited by 193
  37. 2023GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussianscited by 190
  38. 2020Robust Consistent Video Depth Estimationcited by 188

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 89; 2026: 64.

177 papers.

Sort:

By area:

  1. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 431
  2. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 247
  3. 2024MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videoscited by 224
  4. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 163
  5. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  6. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  7. 2024Stereo4D: Learning How Things Move in 3D from Internet Stereo Videoscited by 89
  8. 2025St4RTrack: Simultaneous 4D Reconstruction and Tracking in the Worldcited by 88
  9. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  10. 2024Align3R: Aligned Monocular Depth Estimation for Dynamic Videoscited by 79
  11. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 76
  12. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66
38 more
  1. 2024STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenescited by 53
  2. 2024MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priorscited by 51
  3. 2026InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streamscited by 51
  4. 2024Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videoscited by 48
  5. 20254DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videoscited by 46
  6. 2024SplineGS: Robust Motion-Adaptive Spline for Real-Time Dynamic 3D Gaussians from Monocular Videocited by 44
  7. 2025Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstructioncited by 43
  8. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  9. 2025GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priorscited by 36
  10. 2025Any4D: Unified Feed-Forward Metric 4D Reconstructioncited by 32
  11. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 31
  12. 2025MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splattingcited by 30
  13. 2024Driv3R: Learning Dense 4D Reconstruction for Autonomous Drivingcited by 28
  14. 2025Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Videocited by 27
  15. 2024DAS3R: Dynamics-Aware Gaussian Splatting for Static Scene Reconstructioncited by 25
  16. 2024Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characterscited by 23
  17. 2024ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretrainingcited by 21
  18. 2025FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Videocited by 21
  19. 2025HiMoR: Monocular Deformable Gaussian Reconstruction with Hierarchical Motion Representationcited by 21
  20. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  21. 2024Deblur4DGS: 4D Gaussian Splatting from Blurry Monocular Videocited by 20
  22. 2025MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusioncited by 18
  23. 2025Multi-View 3D Point Trackingcited by 17
  24. 20264RC: 4D Reconstruction via Conditional Querying Anytime and Anywherecited by 17
  25. 2024How Far are AI-generated Videos from Simulating the 3D Visual World: A Learned 3D Evaluation Approachcited by 15
  26. 2025VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Videocited by 15
  27. 2026Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesiscited by 15
  28. 2026Vista4D: Video Reshooting with 4D Point Cloudscited by 15
  29. 2025Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fieldscited by 13
  30. 2025Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videoscited by 13
  31. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  32. 2025L4P: Towards Unified Low-Level 4D Vision Perceptioncited by 12
  33. 2025ReCon-GS: Continuum-Preserved Gaussian Streaming for Fast and Compact Reconstruction of Dynamic Scenescited by 12
  34. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  35. 2024D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Videocited by 11
  36. 2025Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpaintingcited by 11
  37. 2025MoBGS: Motion Deblurring Dynamic 3D Gaussian Splatting for Blurry Monocular Videocited by 11
  38. 2025HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scenecited by 11

50 most cited of 177 All 177 (JSON)