Multi-view Monocular Depth and Uncertainty Prediction with Deep SfM in Dynamic Environments
Not on the map. Why?
Abstract (arXiv)
3D reconstruction of depth and motion from monocular video in dynamic environments is a highly ill-posed problem due to scale ambiguities when projecting to the 2D image domain. In this work, we investigate the performance of the current State-of-the-Art (SotA) deep multi-view systems in such environments. We find that current supervised methods work surprisingly well despite not modelling individual object motions, but make systematic errors due to a lack of dense ground truth data. To detect such errors during usage, we extend the cost volume based Deep Video to Depth (DeepV2D) framework citeteed2018deepv2d with a learned uncertainty. Our Deep Video to certain Depth (DeepV2cD) model allows i) to perform en par or better with current SotA and ii) achieve a better uncertainty measure than the naive Shannon entropy. Our experiments show that a simple filter strategy based on the uncertainty can significantly reduce systematic errors. This results in cleaner reconstructions both on static and dynamic parts of the scene.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 62 of its 88 references that are in the map. First posted on arXiv 21 Jan 2022.
Builds on
62 of 88 references.
References · 62
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2021Vision Transformers for Dense Predictioncited by 1,789
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
- 2015Spatial Transformer Networkscited by 1,423
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
- 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
- 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
- 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
- 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
38 more
- 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
- 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
- 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
- 2018Pyramid Stereo Matching Networkcited by 486
- 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
- 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
- 2016Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescuecited by 368
- 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
- 2020Virtual KITTI 2cited by 345
- 2018GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Posecited by 341
- 2019GA-Net: Guided Aggregation Net for End-to-end Stereo Matchingcited by 242
- 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
- 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
- 2020D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometrycited by 206
- 2016DeMoN: Depth and Motion Network for Learning Monocular Stereocited by 200
- 2021The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depthcited by 190
- 2020Robust Consistent Video Depth Estimationcited by 188
- 2018DeepV2D: Video to Depth with Differentiable Structure from Motioncited by 185
- 2020On the uncertainty of self-supervised monocular depth estimationcited by 182
- 2017DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networkscited by 173
- 2018BA-Net: Dense Bundle Adjustment Networkcited by 173
- 2015Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fieldscited by 169
- 2020Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Cameracited by 166
- 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
- 2015Modelling Uncertainty in Deep Learning for Camera Relocalizationcited by 133
- 2019Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Cameracited by 130
- 2020Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volumecited by 116
- 2017Semi-Supervised Deep Learning for Monocular Depth Map Predictioncited by 109
- 2021TransformerFusion: Monocular RGB Scene Reconstruction using Transformerscited by 107
- 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
- 2018DeepTAM: Deep Tracking and Mappingcited by 87
- 2020Unsupervised Monocular Depth Learning in Dynamic Scenescited by 84
- 2019How do neural networks see depth in single images?cited by 82
- 2019Neural RGB->D Sensing: Depth and Uncertainty from a Video Cameracited by 69
- 2020DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusioncited by 68
- 2020MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Cameracited by 59
- 2021Consistent Depth of Moving Objects in Videocited by 56
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.