# MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection

arXiv: https://arxiv.org/abs/2203.08563 · HTML: https://mlsanity.com/paper/2203.08563/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 40 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Group-wise Correlation Stereo Network](https://mlsanity.com/paper/1903.04025/) (2019)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [Objects are Different: Flexible Monocular 3D Object Detection](https://mlsanity.com/paper/2104.02323/) (2021)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)
- [Stereo R-CNN based 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09738/) (2019)
- [Geometry Uncertainty Projection Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13774/) (2021)
- [MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships](https://mlsanity.com/paper/2003.00504/) (2020)
- [RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving](https://mlsanity.com/paper/2001.03343/) (2020)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [Delving into Localization Errors for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.16237/) (2021)
- [Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving](https://mlsanity.com/paper/1903.11444/) (2019)
- [ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape](https://mlsanity.com/paper/1812.02781/) (2018)
- [Kinematic 3D Object Detection in Monocular Video](https://mlsanity.com/paper/2007.09548/) (2020)
- [Rethinking Pseudo-LiDAR Representation](https://mlsanity.com/paper/2008.04582/) (2020)
- [AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection](https://mlsanity.com/paper/2108.11127/) (2021)
- [Depth-conditioned Dynamic Message Propagation for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.16470/) (2021)
- [Geometry-based Distance Decomposition for Monocular 3D Object Detection](https://mlsanity.com/paper/2104.03775/) (2021)
- [MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation](https://mlsanity.com/paper/2103.12605/) (2021)
- [Monocular 3D Object Detection: An Extrinsic Parameter Free Approach](https://mlsanity.com/paper/2106.15796/) (2021)

## Built on by (most cited first, 30 of 30)

- [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023)
- [EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation](https://mlsanity.com/paper/2303.12787/) (2023)
- [DID-M3D: Decoupling Instance Depth for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.08531/) (2022)
- [MonoCD: Monocular 3D Object Detection with Complementary Depths](https://mlsanity.com/paper/2404.03181/) (2024)
- [Learning Occupancy for Monocular 3D Object Detection](https://mlsanity.com/paper/2305.15694/) (2023)
- [MonoNeRD: NeRF-like Representations for Monocular 3D Object Detection](https://mlsanity.com/paper/2308.09421/) (2023)
- [Densely Constrained Depth Estimator for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10047/) (2022)
- [Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.04448/) (2022)
- [Focal-PETR: Embracing Foreground for Efficient Multi-Camera 3D Object Detection](https://mlsanity.com/paper/2212.05505/) (2022)
- [Weakly Supervised Monocular 3D Object Detection using Multi-View Projection and Direction Consistency](https://mlsanity.com/paper/2303.08686/) (2023)
- [NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization](https://mlsanity.com/paper/2305.17763/) (2023)
- [CARLA Drone: Monocular 3D Object Detection from a Different Perspective](https://mlsanity.com/paper/2408.11958/) (2024)
- [DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking](https://mlsanity.com/paper/2303.16628/) (2023)
- [Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection](https://mlsanity.com/paper/2403.17387/) (2024)
- [SSD-MonoDETR: Supervised Scale-aware Deformable Transformer for Monocular 3D Object Detection](https://mlsanity.com/paper/2305.07270/) (2023)
- [Monocular 3D Object Detection with Bounding Box Denoising in 3D by Perceiver](https://mlsanity.com/paper/2304.01289/) (2023)
- [Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection](https://mlsanity.com/paper/2211.16779/) (2022)
- [3D Video Object Detection with Learnable Object-Centric Global Optimization](https://mlsanity.com/paper/2303.15416/) (2023)
- [OBMO: One Bounding Box Multiple Objects for Monocular 3D Object Detection](https://mlsanity.com/paper/2212.10049/) (2022)
- [Weakly Supervised 3D Object Detection with Multi-Stage Generalization](https://mlsanity.com/paper/2306.05418/) (2023)
- [Vision Foundation Model Driven Foreground-Aware Pseudo-LiDAR Generation for Monocular 3D Object Detection](https://mlsanity.com/paper/2404.09431/) (2024)
- [StereoDETR: Stereo-based Transformer for 3D Object Detection](https://mlsanity.com/paper/2511.18788/) (2025)
- [Towards Fair and Comprehensive Comparisons for Image-Based 3D Object Detection](https://mlsanity.com/paper/2310.05447/) (2023)
- [AuxDepthNet: Real-Time Monocular 3D Object Detection with Depth-Sensitive Features](https://mlsanity.com/paper/2501.03700/) (2025)
- [LeAD-M3D: Leveraging Asymmetric Distillation for Real-Time Monocular 3D Detection](https://mlsanity.com/paper/2512.05663/) (2025)
- [DDStereo: Efficient Dual Decoder Transformers for Stereo 3D Road Anomaly Detection](https://mlsanity.com/paper/2606.24805/) (2026)
- [S3-MonoDETR: Supervised Shape&Scale-perceptive Deformable Transformer for Monocular 3D Object Detection](https://mlsanity.com/paper/2309.00928/) (2023)
- [Efficient Feature Aggregation and Scale-Aware Regression for Monocular 3D Object Detection](https://mlsanity.com/paper/2411.02747/) (2024)
- [MonoVQD: Monocular 3D Object Detection with Variational Query Denoising and Self-Distillation](https://mlsanity.com/paper/2506.14835/) (2025)
- [Mono3DV: Monocular 3D Object Detection with 3D-Aware Bipartite Matching and Variational Query DeNoising](https://mlsanity.com/paper/2601.01036/) (2026)

## Abstract (arXiv)

Due to the inherent ill-posed nature of 2D-3D projection, monocular 3D object detection lacks accurate depth recovery ability. Although the deep neural network (DNN) enables monocular depth-sensing from high-level learned features, the pixel-level cues are usually omitted due to the deep convolution mechanism. To benefit from both the powerful feature representation in DNN and pixel-level geometric constraints, we reformulate the monocular object depth estimation as a progressive refinement problem and propose a joint semantic and geometric cost volume to model the depth error. Specifically, we first leverage neural networks to learn the object position, dimension, and dense normalized 3D object coordinates. Based on the object depth, the dense coordinates patch together with the corresponding object features is reprojected to the image space to build a cost volume in a joint semantic and geometric error manner. The final depth is obtained by feeding the cost volume to a refinement network, where the distribution of semantic and geometric error is regularized by direct depth supervision. Through effectively mitigating depth error by the refinement framework, we achieve state-of-the-art results on both the KITTI and Waymo datasets.