Towards Continual, Online, Self-Supervised Depth
Not on the map. Why?
Abstract (arXiv)
Although depth extraction with passive sensors has seen remarkable improvement with deep learning, these approaches may fail to obtain correct depth if they are exposed to environments not observed during training. Online adaptation, where the neural network trains while deployed, with self-supervised learning provides a convenient solution as the network can learn from the scene where it is deployed without external supervision. However, online adaptation causes a neural network to forget the past. Thus, past training is wasted and the network is not able to provide good results if it observes past scenes. This work deals with practical online-adaptation where the input is online and temporally-correlated, and training is completely self-supervised. Regularization and replay-based methods without task boundaries are proposed to avoid catastrophic forgetting while adapting to online data. Effort has been made to make the proposed approach suitable for practical use. We apply our method to both structure-from-motion and stereo depth estimation. We evaluate our method on diverse public datasets that include outdoor, indoor and synthetic scenes. Qualitative and quantitative results with both structure-from-motion and stereo show superior forgetting as well as adaptation performance compared to recent methods. Furthermore, the proposed method incurs negligible overhead compared to fine-tuning for online adaptation, proving to be an adequate choice in terms of plasticity, stability and applicability. The proposed approach is more inline with the artificial general intelligence paradigm as the neural network learns continually with no supervision. Source code is available at https://github.com/umarKarim/cou_sfm and https://github.com/umarKarim/cou_stereo.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 28 Feb 2021.
Builds on
49 of 68 references.
References · 49
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2016Learning without Forgettingcited by 2,496
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
- 2015Spatial Transformer Networkscited by 1,429
- 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
- 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
- 2017Continual Learning with Deep Generative Replaycited by 1,158
- 2016Pruning Filters for Efficient ConvNetscited by 1,143
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
37 more
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
- 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 820
- 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 782
- 2019Gradient based sample selection for online continual learningcited by 571
- 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
- 2016Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescuecited by 368
- 2020Virtual KITTI 2cited by 353
- 2019Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Videocited by 244
- 2018Task-Free Continual Learningcited by 221
- 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
- 2019Meta-Learning Representations for Continual Learningcited by 189
- 2020On the uncertainty of self-supervised monocular depth estimationcited by 182
- 2020Understanding the Role of Training Regimes in Continual Learningcited by 180
- 2019Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Camerascited by 170
- 2019Continual Unsupervised Representation Learningcited by 147
- 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 134
- 2019Self-Supervised Monocular Depth Hintscited by 127
- 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
- 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
- 2019Latent Replay for Real-Time Continual Learningcited by 97
- 2018Real-time self-adaptive deep stereocited by 80
- 2020Incremental Learning In Online Scenariocited by 71
- 2018T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Taskscited by 70
- 2020Self-Supervised Monocular Scene Flow Estimationcited by 64
- 2020DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised Representation Learningcited by 60
- 2020La-MAML: Look-ahead Meta Learning for Continual Learningcited by 52
- 2020Forget About the LiDAR: Self-Supervised Depth Estimators with MED Probability Volumescited by 50
- 2020Online Continual Learning under Extreme Memory Constraintscited by 48
- 2018Unsupervised Adversarial Depth Estimation using Cycled Generative Networkscited by 45
- 2019Learning to Adapt for Stereocited by 42
- 2020Pseudo RGB-D for Self-Improving Monocular SLAM and Depth Predictioncited by 41
- 2020P2Net: Patch-match and Plane-regularization for Unsupervised Indoor Depth Estimationcited by 41
- 2020Wasserstein Distances for Stereo Disparity Estimationcited by 38
- 2020Bi3D: Stereo Depth Estimation via Binary Classificationscited by 34
- 2019Moving Indoor: Unsupervised Video Depth Learning in Challenging Environmentscited by 31
- 2020Depth Sensing Beyond LiDAR Rangecited by 10
- 2020Visual Memorability for Robotic Interestingness via Unsupervised Online Learningcited by 8
Built on by
Citing papers per year: 2023: 1.
1 paper.