mlsanityData to 14 Oct 2026

Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos

arXiv 2412.09621 ↗First posted 12 Dec 2024cs.CVcited by 89

Not on the map. Why?

Abstract (arXiv)

Learning to understand dynamic 3D scenes from imagery is crucial for applications ranging from robotics to scene reconstruction. Yet, unlike other problems where large-scale supervised training has enabled rapid progress, directly supervising methods for recovering 3D motion remains challenging due to the fundamental difficulty of obtaining ground truth annotations. We present a system for mining high-quality 4D reconstructions from internet stereoscopic, wide-angle videos. Our system fuses and filters the outputs of camera pose estimation, stereo depth estimation, and temporal tracking methods into high-quality dynamic 3D reconstructions. We use this method to generate large-scale data in the form of world-consistent, pseudo-metric 3D point clouds with long-term motion trajectories. We demonstrate the utility of this data by training a variant of DUSt3R to predict structure and 3D motion from real-world image pairs, showing that training on our reconstructed data enables generalization to diverse real-world scenes. Project page and data at: https://stereo4d.github.io

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Builds on 83 of its 119 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

83 of 119 references.

References · 83

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  6. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  7. 2021Vision Transformers for Dense Predictioncited by 1,803
  8. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  9. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  10. 2024Depth Anything V2cited by 1,422
  11. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
38 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  2. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  3. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 988
  4. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  5. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 881
  6. 2020ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMcited by 878
  7. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  8. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  9. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  10. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  11. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  12. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  13. 2018Pyramid Stereo Matching Networkcited by 489
  14. 2016Direct Sparse Odometrycited by 485
  15. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  16. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  17. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  18. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 382
  19. 2022Kubric: A scalable dataset generatorcited by 381
  20. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 365
  21. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 360
  22. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
  23. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
  24. 2019Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inferencecited by 254
  25. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 250
  26. 2019GA-Net: Guided Aggregation Net for End-to-end Stereo Matchingcited by 244
  27. 2024MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videoscited by 228
  28. 2023Visual Geometry Grounded Deep Structure From Motioncited by 227
  29. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  30. 2020Consistent Video Depth Estimationcited by 220
  31. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 217
  32. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
  33. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  34. 2021Pixel-Perfect Structure-from-Motion with Featuremetric Refinementcited by 200
  35. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 200
  36. 2023Robust Dynamic Radiance Fieldscited by 194
  37. 2023NeRSemble: Multi-view Radiance Field Reconstruction of Human Headscited by 194
  38. 2023COLMAP-Free 3D Gaussian Splattingcited by 193

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 38.

89 papers.

Sort:

By area:

  1. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 170
  2. 2025Easi3R: Estimating Disentangled Motion from DUSt3R Without Trainingcited by 91
  3. 2025St4RTrack: Simultaneous 4D Reconstruction and Tracking in the Worldcited by 89
  4. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 87
  5. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 77
  6. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 77
  7. 2025TAPIP3D: Tracking Any Point in Persistent 3D Geometrycited by 69
  8. 2025Towards Understanding Camera Motions in Any Videocited by 64
  9. 2025Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstructioncited by 44
  10. 2026LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstructioncited by 37
  11. 2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 35
  12. 2025Any4D: Unified Feed-Forward Metric 4D Reconstructioncited by 35
38 more
  1. 2025Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matchingcited by 28
  2. 2025Dynamic Camera Poses and Where to Find Themcited by 27
  3. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 27
  4. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 19
  5. 2025Multi-View 3D Point Trackingcited by 17
  6. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 14
  7. 2026WildDet3D: Scaling Promptable 3D Detection in the Wildcited by 14
  8. 2026Track4World: Feedforward World-centric Dense 3D Tracking of All Pixelscited by 13
  9. 2026Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Videocited by 12
  10. 2025Rig3R: Rig-Aware Conditioning for Learned 3D Reconstructioncited by 11
  11. 2025Alligat0R: Pre-Training Through Co-Visibility Segmentation for Relative Camera Pose Regressioncited by 10
  12. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  13. 2025M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversioncited by 10
  14. 20254D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimationcited by 10
  15. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  16. 2025UniGeo: Taming Video Diffusion for Unified Consistent Geometry Estimationcited by 9
  17. 2026CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoningcited by 9
  18. 2026MoRe: Motion-aware Feed-forward 4D Reconstruction Transformercited by 9
  19. 2025LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoningcited by 8
  20. 2025PanoSplatt3R: Leveraging Perspective Pretraining for Generalized Unposed Wide-Baseline Panorama Reconstructioncited by 8
  21. 2025PAGE-4D: Disentangled pose and geometry estimation for vggt-4d perceptioncited by 8
  22. 2025Understanding Multi-View Transformerscited by 8
  23. 2025E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Modelscited by 7
  24. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Modelscited by 7
  25. 2025TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixelscited by 7
  26. 2025Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesiscited by 6
  27. 2025LASER: Layer-wise Scale Alignment for Training-Free Streaming 4D Reconstructioncited by 6
  28. 2026LabelAny3D: Label Any Object 3D in the Wildcited by 6
  29. 2025A Scene is Worth a Thousand Features: Feed-Forward Camera Localization from a Collection of Image Featurescited by 5
  30. 2025DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modelingcited by 5
  31. 2025WorldReel: 4D Video Generation with Consistent Geometry and Motion Modelingcited by 5
  32. 2026UFO-4D: Unposed Feedforward 4D Reconstruction from Two Imagescited by 5
  33. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  34. 2026TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Trackingcited by 5
  35. 2026MolmoMotion: Forecasting Point Trajectories in 3D with Language Instructioncited by 5
  36. 2026SM4RT: Learning Structured Motion Geometry for 4D Reconstructioncited by 5
  37. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  38. 2025Lite Any Stereo: Efficient Zero-Shot Stereo Matchingcited by 4

50 most cited of 89 All 89 (JSON)