mlsanityData to 7 Oct 2026

SfM-Net: Learning of Structure and Motion from Video

arXiv 1704.07804 ↗First posted 25 Apr 2017cs.CVcited by 89

Not on the map. Why?

Abstract (arXiv)

We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates. The model can be trained with various degrees of supervision: 1) self-supervised by the re-projection photometric error (completely unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth estimates and successfully estimates frame-to-frame camera rotations and translations. It often successfully segments the moving objects in the scene, even though such supervision is never provided.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 21; 2023: 16; 2024: 13; 2025: 5; 2026: 3.

89 papers.

Sort:

By area:

  1. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  2. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  3. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  4. 2021Is Pseudo-Lidar needed for Monocular 3D Object detection?cited by 248
  5. 2021The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depthcited by 190
  6. 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 138
  7. 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
  8. 2022RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wildcited by 102
  9. 2021NeuTex: Neural Texture Mapping for Volumetric Neural Renderingcited by 98
  10. 2023Audio-Visual Segmentation with Semanticscited by 73
  11. 2023Detector-Free Structure from Motioncited by 71
  12. 2021Unsupervised Scale-consistent Depth Learning from Videocited by 69
38 more
  1. 2023Tetra-NeRF: Representing Neural Radiance Fields Using Tetrahedracited by 57
  2. 2021Semantics for Robotic Mapping, Perception and Interaction: A Surveycited by 56
  3. 2021Radar-Camera Pixel Depth Association for Depth Completioncited by 49
  4. 2021UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Renderingcited by 44
  5. 2023Egocentric Audio-Visual Object Localizationcited by 40
  6. 2022BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimationcited by 35
  7. 2022Learning Optical Flow, Depth, and Scene Flow without Real-World Labelscited by 31
  8. 2022PVO: Panoptic Visual Odometrycited by 26
  9. 2022EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queriescited by 25
  10. 2023NeuManifold: Neural Watertight Manifold Reconstruction with Efficient and High-Quality Rendering Supportcited by 25
  11. 2023SQLdepth: Generalizable Self-Supervised Fine-Structured Monocular Depth Estimationcited by 21
  12. 2023Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenescited by 20
  13. 2021Outdoor inverse rendering from a single image using multiview self-supervisioncited by 18
  14. 2021Learning monocular 3D reconstruction of articulated categories from motioncited by 18
  15. 2021Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Losscited by 16
  16. 2021Deep Permutation Equivariant Structure from Motioncited by 15
  17. 2021Pushing it out of the Way: Interactive Visual Navigationcited by 14
  18. 2022Maximizing Self-supervision from Thermal Image for Effective Self-supervised Learning of Depth and Ego-motioncited by 14
  19. 2022ChiTransformer:Towards Reliable Stereo from Cuescited by 14
  20. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13
  21. 2021Dimensions of Motion: Monocular Prediction through Flow Subspacescited by 10
  22. 2022MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Posecited by 10
  23. 2023EC-SfM: Efficient Covisibility-based Structure-from-Motion for Both Sequential and Unordered Imagescited by 9
  24. 2021Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenescited by 8
  25. 2021CamLessMonoDepth: Monocular Depth Estimation with Unknown Camera Parameterscited by 8
  26. 2024SPIdepth: Strengthened Pose Information for Self-supervised Monocular Depth Estimationcited by 8
  27. 2024Cross-spectral Gated-RGB Stereo Depth Estimationcited by 8
  28. 2021On the Sins of Image Synthesis Loss for Self-supervised Depth Estimationcited by 7
  29. 2021A Survey and Empirical Evaluation of Parallel Deep Learning Frameworkscited by 7
  30. 2021ColDE: A Depth Estimation Framework for Colonoscopy Reconstructioncited by 7
  31. 2022Vision-Based Environmental Perception for Autonomous Drivingcited by 7
  32. 2023Gated Stereo: Joint Depth Estimation from Gated and Wide-Baseline Active Stereo Cuescited by 7
  33. 2024Ego3DT: Tracking Every 3D Object in Ego-centric Videoscited by 7
  34. 2021Gated2Gated: Self-Supervised Depth Estimation from Gated Imagescited by 6
  35. 2023MonoProb: Self-Supervised Monocular Depth Estimation with Interpretable Uncertaintycited by 6
  36. 2024ProDepth: Boosting Self-Supervised Multi-Frame Monocular Depth with Probabilistic Fusioncited by 6
  37. 2024GroCo: Ground Constraint for Metric Self-Supervised Monocular Depthcited by 6
  38. 2024On Moving Object Segmentation from Monocular Video with Transformerscited by 6

50 most cited of 89 All 89 (JSON)