# Self-supervised Learning of 3D Object Understanding by Data Association and Landmark Estimation for Image Sequence

arXiv: https://arxiv.org/abs/2104.07077 · HTML: https://mlsanity.com/paper/2104.07077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 30 of 34)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Domain Adaptive Faster R-CNN for Object Detection in the Wild](https://mlsanity.com/paper/1803.03243/) (2018)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [SESS: Self-Ensembling Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/1912.11803/) (2019)
- [SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans](https://mlsanity.com/paper/1912.00036/) (2019)
- [Self-EMD: Self-Supervised Object Detection without ImageNet](https://mlsanity.com/paper/2011.13677/) (2020)
- [Viewpoints and Keypoints](https://mlsanity.com/paper/1411.6067/) (2014)
- [Improving Object Detection with Selective Self-supervised Self-training](https://mlsanity.com/paper/2007.09162/) (2020)
- [EAO-SLAM: Monocular Semi-Dense Object SLAM Based on Ensemble Data Association](https://mlsanity.com/paper/2004.12730/) (2020)
- [Monocular Object and Plane SLAM in Structured Environments](https://mlsanity.com/paper/1809.03415/) (2018)
- [Efficient Learning of Domain-invariant Image Representations](https://mlsanity.com/paper/1301.3224/) (2013)
- [Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection](https://mlsanity.com/paper/1604.04693/) (2016)
- [Learning Object-specific Distance from a Monocular Image](https://mlsanity.com/paper/1909.04182/) (2019)
- [Constructing Category-Specific Models for Monocular Object-SLAM](https://mlsanity.com/paper/1802.09292/) (2018)

## Abstract (arXiv)

In this paper, we propose a self-supervised learningmethod for multi-object pose estimation. 3D object under-standing from 2D image is a challenging task that infers ad-ditional dimension from reduced-dimensional information.In particular, the estimation of the 3D localization or orien-tation of an object requires precise reasoning, unlike othersimple clustering tasks such as object classification. There-fore, the scale of the training dataset becomes more cru-cial. However, it is challenging to obtain large amount of3D dataset since achieving 3D annotation is expensive andtime-consuming. If the scale of the training dataset can beincreased by involving the image sequence obtained fromsimple navigation, it is possible to overcome the scale lim-itation of the dataset and to have efficient adaptation tothe new environment. However, when the self annotation isconducted on single image by the network itself, trainingperformance of the network is bounded to the self perfor-mance. Therefore, we propose a strategy to exploit multipleobservations of the object in the image sequence in orderto surpass the self-performance: first, the landmarks for theglobal object map are estimated through network predic-tion and data association, and the corrected annotation fora single frame is obtained. Then, network fine-tuning is con-ducted including the dataset obtained by self-annotation,thereby exceeding the performance boundary of the networkitself. The proposed method was evaluated on the KITTIdriving scene dataset, and we demonstrate the performanceimprovement in the pose estimation of multi-object in 3D space.