# GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection

arXiv: https://arxiv.org/abs/2103.17202 · HTML: https://mlsanity.com/paper/2103.17202/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 91 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 30 of 59)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Differentiation of Blackbox Combinatorial Solvers](https://mlsanity.com/paper/1912.02175/) (2019)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [Stereo R-CNN based 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09738/) (2019)
- [MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships](https://mlsanity.com/paper/2003.00504/) (2020)
- [RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving](https://mlsanity.com/paper/2001.03343/) (2020)
- [Probabilistic Anchor Assignment with IoU Prediction for Object Detection](https://mlsanity.com/paper/2007.08103/) (2020)
- [Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving](https://mlsanity.com/paper/1903.11444/) (2019)
- [ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape](https://mlsanity.com/paper/1812.02781/) (2018)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [Kinematic 3D Object Detection in Monocular Video](https://mlsanity.com/paper/2007.09548/) (2020)
- [Learning non-maximum suppression](https://mlsanity.com/paper/1705.02950/) (2017)

## Built on by (most cited first, 30 of 66)

- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection](https://mlsanity.com/paper/2203.17054/) (2022)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer](https://mlsanity.com/paper/2203.10981/) (2022)
- [AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection](https://mlsanity.com/paper/2108.11127/) (2021)
- [MonoDistill: Learning Spatial Features for Monocular 3D Object Detection](https://mlsanity.com/paper/2201.10830/) (2022)
- [Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detection](https://mlsanity.com/paper/2112.04628/) (2021)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [AutoRF: Learning 3D Object Radiance Fields from Single View Observations](https://mlsanity.com/paper/2204.03593/) (2022)
- [Polar Parametrization for Vision-based Surround-View 3D Detection](https://mlsanity.com/paper/2206.10965/) (2022)
- [Progressive Coordinate Transforms for Monocular 3D Object Detection](https://mlsanity.com/paper/2108.05793/) (2021)
- [Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2205.14951/) (2022)
- [DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.10758/) (2022)
- [Diversity Matters: Fully Exploiting Depth Clues for Reliable Monocular 3D Object Detection](https://mlsanity.com/paper/2205.09373/) (2022)
- [DID-M3D: Decoupling Instance Depth for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.08531/) (2022)
- [MonoGround: Detecting Monocular 3D Objects from the Ground](https://mlsanity.com/paper/2206.07372/) (2022)
- [Learning Occupancy for Monocular 3D Object Detection](https://mlsanity.com/paper/2305.15694/) (2023)
- [Tame a Wild Camera: In-the-Wild Monocular Camera Calibration](https://mlsanity.com/paper/2306.10988/) (2023)
- [GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2310.08261/) (2023)
- [Pseudo-Stereo for Monocular 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.02112/) (2022)
- [MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection](https://mlsanity.com/paper/2203.08563/) (2022)
- [Lidar Point Cloud Guided Monocular 3D Object Detection](https://mlsanity.com/paper/2104.09035/) (2021)
- [Mix-Teaching: A Simple, Unified and Effective Semi-Supervised Learning Framework for Monocular 3D Object Detection](https://mlsanity.com/paper/2207.04448/) (2022)
- [Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13931/) (2021)
- [Voxel-based 3D Detection and Reconstruction of Multiple Objects from a Single Image](https://mlsanity.com/paper/2111.03098/) (2021)
- [MonoLSS: Learnable Sample Selection For Monocular 3D Detection](https://mlsanity.com/paper/2312.14474/) (2023)
- [MultiCorrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detection](https://mlsanity.com/paper/2402.11677/) (2024)
- [Differentiable Clustering with Perturbed Spanning Forests](https://mlsanity.com/paper/2305.16358/) (2023)

## Abstract (arXiv)

Modern 3D object detectors have immensely benefited from the end-to-end learning idea. However, most of them use a post-processing algorithm called Non-Maximal Suppression (NMS) only during inference. While there were attempts to include NMS in the training pipeline for tasks such as 2D object detection, they have been less widely adopted due to a non-mathematical expression of the NMS. In this paper, we present and integrate GrooMeD-NMS -- a novel Grouped Mathematically Differentiable NMS for monocular 3D object detection, such that the network is trained end-to-end with a loss on the boxes after NMS. We first formulate NMS as a matrix operation and then group and mask the boxes in an unsupervised manner to obtain a simple closed-form expression of the NMS. GrooMeD-NMS addresses the mismatch between training and inference pipelines and, therefore, forces the network to select the best 3D box in a differentiable manner. As a result, GrooMeD-NMS achieves state-of-the-art monocular 3D object detection results on the KITTI benchmark dataset performing comparably to monocular video-based methods. Code and models at https://github.com/abhi1kumar/groomed_nms