# Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview

arXiv: https://arxiv.org/abs/2105.14291 · HTML: https://mlsanity.com/paper/2105.14291/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 123 of its 187 references that are in the map.
- First posted on arXiv 29 May 2021.

## Builds on (most cited first, 30 of 123)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)

## Built on by (most cited first, 30 of 31)

- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [Bridging the Domain Gap for Multi-Agent Perception](https://mlsanity.com/paper/2210.08451/) (2022)
- [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023)
- [Deep Learning-Based Object Pose Estimation: A Comprehensive Survey](https://mlsanity.com/paper/2405.07801/) (2024)
- [Object Level Depth Reconstruction for Category Level 6D Object Pose Estimation From Monocular RGB Image](https://mlsanity.com/paper/2204.01586/) (2022)
- [Towards Bridging the Space Domain Gap for Satellite Pose Estimation using Event Sensing](https://mlsanity.com/paper/2209.11945/) (2022)
- [Challenges for Monocular 6D Object Pose Estimation in Robotics](https://mlsanity.com/paper/2307.12172/) (2023)
- [ACR-Pose: Adversarial Canonical Representation Reconstruction Network for Category Level 6D Object Pose Estimation](https://mlsanity.com/paper/2111.10524/) (2021)
- [FvOR: Robust Joint Shape and Pose Optimization for Few-view Object Reconstruction](https://mlsanity.com/paper/2205.07763/) (2022)
- [Global Adaptation meets Local Generalization: Unsupervised Domain Adaptation for 3D Human Pose Estimation](https://mlsanity.com/paper/2303.16456/) (2023)
- [MonoCInIS: Camera Independent Monocular 3D Object Detection using Instance Segmentation](https://mlsanity.com/paper/2110.00464/) (2021)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [SA6D: Self-Adaptive Few-Shot 6D Pose Estimator for Novel and Occluded Objects](https://mlsanity.com/paper/2308.16528/) (2023)
- [PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features](https://mlsanity.com/paper/2312.02781/) (2023)
- [PRISM-DP: Spatial Pose-based Observations for Diffusion-Policies via Segmentation, Mesh Generation, and Pose Tracking](https://mlsanity.com/paper/2504.20359/) (2025)
- [Category-Agnostic 6D Pose Estimation with Conditional Neural Processes](https://mlsanity.com/paper/2206.07162/) (2022)
- [Certifying Robustness of Learning-Based Keypoint Detection and Pose Estimation Methods](https://mlsanity.com/paper/2408.00117/) (2024)
- [From Orthomosaics to Raw UAV Imagery: Enhancing Palm Detection and Crown-Center Localization](https://mlsanity.com/paper/2509.12400/) (2025)
- [Open Challenges for Monocular Single-shot 6D Object Pose Estimation](https://mlsanity.com/paper/2302.11827/) (2023)
- [IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools](https://mlsanity.com/paper/2602.05555/) (2026)
- [FuRPE: Learning Full-body Reconstruction from Part Experts](https://mlsanity.com/paper/2212.00731/) (2022)
- [Deep Learning for Human Parsing: A Survey](https://mlsanity.com/paper/2301.12416/) (2023)
- [From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers](https://mlsanity.com/paper/2406.07785/) (2024)
- [FAFA: Frequency-Aware Flow-Aided Self-Supervision for Underwater Object Pose Estimation](https://mlsanity.com/paper/2409.16600/) (2024)
- [6DAttack: Backdoor Attacks in the 6DoF Pose Estimation](https://mlsanity.com/paper/2512.19058/) (2025)
- [MonoSIM: Simulating Learning Behaviors of Heterogeneous Point Cloud Object Detectors for Monocular 3D Object Detection](https://mlsanity.com/paper/2208.09446/) (2022)
- [KVN: Keypoints Voting Network with Differentiable RANSAC for Stereo Pose Estimation](https://mlsanity.com/paper/2307.11543/) (2023)
- [Category-Level and Open-Set Object Pose Estimation for Robotics](https://mlsanity.com/paper/2504.19572/) (2025)
- [Consensus-Driven Uncertainty for Robotic Grasping based on RGB Perception](https://mlsanity.com/paper/2506.20045/) (2025)

## Abstract (arXiv)

Object pose detection and tracking has recently attracted increasing attention due to its wide applications in many areas, such as autonomous driving, robotics, and augmented reality. Among methods for object pose detection and tracking, deep learning is the most promising one that has shown better performance than others. However, survey study about the latest development of deep learning-based methods is lacking. Therefore, this study presents a comprehensive review of recent progress in object pose detection and tracking that belongs to the deep learning technical route. To achieve a more thorough introduction, the scope of this study is limited to methods taking monocular RGB/RGBD data as input and covering three kinds of major tasks: instance-level monocular object pose detection, category-level monocular object pose detection, and monocular object pose tracking. In our work, metrics, datasets, and methods of both detection and tracking are presented in detail. Comparative results of current state-of-the-art methods on several publicly available datasets are also presented, together with insightful observations and inspiring future research directions.