# Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations

arXiv: https://arxiv.org/abs/2403.01172 · HTML: https://mlsanity.com/paper/2403.01172/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://mlsanity.com/paper/2209.09858/) (2022)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [A Survey of Modern Deep Learning based Object Detection Models](https://mlsanity.com/paper/2104.11892/) (2021)
- [Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild](https://mlsanity.com/paper/2203.03800/) (2022)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Open-set 3D Object Detection](https://mlsanity.com/paper/2112.01135/) (2021)
- [Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Cars](https://mlsanity.com/paper/1707.00051/) (2017)
- [Uncertainty for Identifying Open-Set Errors in Visual Object Detection](https://mlsanity.com/paper/2104.01328/) (2021)
- [Out-of-Distribution Detection for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2209.14435/) (2022)
- [Monitoring and Diagnosability of Perception Systems](https://mlsanity.com/paper/2011.07010/) (2020)
- [Monitoring of Perception Systems: Deterministic, Probabilistic, and Learning-based Fault Detection and Identification](https://mlsanity.com/paper/2205.10906/) (2022)
- [Online Monitoring of Object Detection Performance During Deployment](https://mlsanity.com/paper/2011.07750/) (2020)
- [Post-hoc Models for Performance Estimation of Machine Learning Inference](https://mlsanity.com/paper/2110.02459/) (2021)
- [Pixel Invisibility: Detecting Objects Invisible in Color Images](https://mlsanity.com/paper/2006.08383/) (2020)

## Built on by (most cited first, 6 of 6)

- [Run-time Monitoring of 3D Object Detection in Automated Driving Systems Using Early Layer Neural Activation Patterns](https://mlsanity.com/paper/2404.07685/) (2024)
- [Integrity Monitoring of 3D Object Detection in Automated Driving Systems using Raw Activation Patterns and Spatial Filtering](https://mlsanity.com/paper/2405.07600/) (2024)
- [Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurance](https://mlsanity.com/paper/2511.08439/) (2025)
- [A 360-degree Multi-camera System for Blue Emergency Light Detection Using Color Attention RT-DETR and the ABLDataset](https://mlsanity.com/paper/2603.05058/) (2026)
- [LFA: Layer Feature Attention for Run-Time Introspection of 2D Object Detectors in Automated Driving](https://mlsanity.com/paper/2606.00372/) (2026)
- [Auditing Latent-Space Monitors for Autonomous Driving](https://mlsanity.com/paper/2609.30557/) (2026)

## Abstract (arXiv)

Reliable detection of various objects and road users in the surrounding environment is crucial for the safe operation of automated driving systems (ADS). Despite recent progresses in developing highly accurate object detectors based on Deep Neural Networks (DNNs), they still remain prone to detection errors, which can lead to fatal consequences in safety-critical applications such as ADS. An effective remedy to this problem is to equip the system with run-time monitoring, named as introspection in the context of autonomous systems. Motivated by this, we introduce a novel introspection solution, which operates at the frame level for DNN-based 2D object detection and leverages neural network activation patterns. The proposed approach pre-processes the neural activation patterns of the object detector's backbone using several different modes. To provide extensive comparative analysis and fair comparison, we also adapt and implement several state-of-the-art (SOTA) introspection mechanisms for error detection in 2D object detection, using one-stage and two-stage object detectors evaluated on KITTI and BDD datasets. We compare the performance of the proposed solution in terms of error detection, adaptability to dataset shift, and, computational and memory resource requirements. Our performance evaluation shows that the proposed introspection solution outperforms SOTA methods, achieving an absolute reduction in the missed error ratio of 9% to 17% in the BDD dataset.