mlsanityData to 14 Oct 2026

Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking

arXiv 2206.03666 ↗First posted 8 Jun 2022cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving. Approaches to monocular 3D perception including detection and tracking, however, often yield inferior performance when compared to LiDAR-based techniques. Through systematic analysis, we identified that per-object depth estimation accuracy is a major factor bounding the performance. Motivated by this observation, we propose a multi-level fusion method that combines different representations (RGB and pseudo-LiDAR) and temporal information across multiple frames for objects (tracklets) to enhance per-object depth estimation. Our proposed fusion method achieves the state-of-the-art performance of per-object depth estimation on the Waymo Open Dataset, the KITTI detection dataset, and the KITTI MOT dataset. We further demonstrate that by simply replacing estimated depth with fusion-enhanced depth, we can achieve significant improvements in monocular 3D perception tasks, including detection and tracking.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

46 of 55 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  4. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  5. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  6. 2019Objects as Pointscited by 1,175
  7. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  8. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  9. 2016Simple Online and Realtime Trackingcited by 868
  10. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 820
  11. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  12. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
34 more
  1. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 490
  2. 2020Tracking Objects as Pointscited by 483
  3. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  4. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  5. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  6. 2019Towards Real-Time Multi-Object Trackingcited by 311
  7. 2020Deep Continuous Fusion for Multi-Sensor 3D Object Detectioncited by 263
  8. 2019M3D-RPN: Monocular 3D Region Proposal Network for Object Detectioncited by 252
  9. 2018FlowNet3D: Learning Scene Flow in 3D Point Cloudscited by 242
  10. 2019Disentangling Monocular 3D Object Detectioncited by 217
  11. 2020SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimationcited by 204
  12. 2020Multi-Task Multi-Sensor Fusion for 3D Object Detectioncited by 203
  13. 2020On the uncertainty of self-supervised monocular depth estimationcited by 182
  14. 2017Flow-Guided Feature Aggregation for Video Object Detectioncited by 177
  15. 2019Learning Depth-Guided Convolutions for Monocular 3D Object Detectioncited by 169
  16. 2020MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationshipscited by 153
  17. 2020RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Drivingcited by 150
  18. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 146
  19. 2019Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Drivingcited by 137
  20. 2020Semantically-Guided Representation Learning for Self-Supervised Monocular Depthcited by 134
  21. 2018High Quality Monocular Depth Estimation via Transfer Learningcited by 133
  22. 2019Monocular 3D Object Detection with Pseudo-LiDAR Point Cloudcited by 113
  23. 2020GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learningcited by 113
  24. 2020Rethinking Pseudo-LiDAR Representationcited by 109
  25. 2018Joint Monocular 3D Vehicle Detection and Trackingcited by 77
  26. 2020RadarNet: Exploiting Radar for Robust Perception of Dynamic Objectscited by 48
  27. 2019Context R-CNN: Long Term Temporal Context for Per-Camera Object Detectioncited by 45
  28. 2020Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Drivingcited by 41
  29. 2019Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Losscited by 30
  30. 2019RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Drivingcited by 23
  31. 2019Looking Fast and Slow: Memory-Guided Mobile Video Object Detectioncited by 22
  32. 2019Task-Aware Monocular Depth Estimation for 3D Object Detectioncited by 22
  33. 2019Learning Object-specific Distance from a Monocular Imagecited by 15
  34. 2020SoDA: Multi-Object Tracking with Soft Data Associationcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 4.

9 papers.

  1. 2022Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detectioncited by 178
  2. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  3. 2023STEPS: Joint Self-supervised Nighttime Image Enhancement and Depth Estimationcited by 37
  4. 2023NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localizationcited by 13
  5. 20233D Multiple Object Tracking on Autonomous Driving: A Literature Reviewcited by 4
  6. 2024You Only Look Bottom-Up for Monocular 3D Object Detectioncited by 3
  7. 2024Robust Inverse Graphics via Probabilistic Inferencecited by 1
  8. 2024Compass: A Decentralized Scheduler for Latency-Sensitive ML Workflowscited by 0
  9. 2024Enhanced Object Tracking by Self-Supervised Auxiliary Depth Estimation Learningcited by 0