# Run-time Monitoring of 3D Object Detection in Automated Driving Systems Using Early Layer Neural Activation Patterns

arXiv: https://arxiv.org/abs/2404.07685 · HTML: https://mlsanity.com/paper/2404.07685/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 21 of 21)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://mlsanity.com/paper/2209.09858/) (2022)
- [Eigen-CAM: Class Activation Map using Principal Components](https://mlsanity.com/paper/2008.00299/) (2020)
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [Cooperative Perception for 3D Object Detection in Driving Scenarios using Infrastructure Sensors](https://mlsanity.com/paper/1912.12147/) (2019)
- [Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection](https://mlsanity.com/paper/1804.05132/) (2018)
- [Can We Trust You? On Calibration of a Probabilistic Object Detector for Autonomous Driving](https://mlsanity.com/paper/1909.12358/) (2019)
- [Open-set 3D Object Detection](https://mlsanity.com/paper/2112.01135/) (2021)
- [Failing to Learn: Autonomously Identifying Perception Failures for Self-driving Cars](https://mlsanity.com/paper/1707.00051/) (2017)
- [Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends](https://mlsanity.com/paper/2101.01364/) (2021)
- [Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection](https://mlsanity.com/paper/2012.12195/) (2020)
- [Out-of-Distribution Detection for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2209.14435/) (2022)
- [Did You Miss the Sign? A False Negative Alarm System for Traffic Sign Detectors](https://mlsanity.com/paper/1903.06391/) (2019)
- [Monitoring and Diagnosability of Perception Systems](https://mlsanity.com/paper/2011.07010/) (2020)
- [Monitoring of Perception Systems: Deterministic, Probabilistic, and Learning-based Fault Detection and Identification](https://mlsanity.com/paper/2205.10906/) (2022)
- [Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations](https://mlsanity.com/paper/2403.01172/) (2024)
- [Post-hoc Models for Performance Estimation of Machine Learning Inference](https://mlsanity.com/paper/2110.02459/) (2021)

## Built on by (most cited first, 3 of 3)

- [Runtime Safety Monitoring of Deep Neural Networks for Perception: A Survey](https://mlsanity.com/paper/2511.05982/) (2025)
- [Integrity Monitoring of 3D Object Detection in Automated Driving Systems using Raw Activation Patterns and Spatial Filtering](https://mlsanity.com/paper/2405.07600/) (2024)
- [Knowledge-Guided Failure Prediction: Detecting When Object Detectors Miss Safety-Critical Objects](https://mlsanity.com/paper/2603.25499/) (2026)

## Abstract (arXiv)

Monitoring the integrity of object detection for errors within the perception module of automated driving systems (ADS) is paramount for ensuring safety. Despite recent advancements in deep neural network (DNN)-based object detectors, their susceptibility to detection errors, particularly in the less-explored realm of 3D object detection, remains a significant concern. State-of-the-art integrity monitoring (also known as introspection) mechanisms in 2D object detection mainly utilise the activation patterns in the final layer of the DNN-based detector's backbone. However, that may not sufficiently address the complexities and sparsity of data in 3D object detection. To this end, we conduct, in this article, an extensive investigation into the effects of activation patterns extracted from various layers of the backbone network for introspecting the operation of 3D object detectors. Through a comparative analysis using Kitti and NuScenes datasets with PointPillars and CenterPoint detectors, we demonstrate that using earlier layers' activation patterns enhances the error detection performance of the integrity monitoring system, yet increases computational complexity. To address the real-time operation requirements in ADS, we also introduce a novel introspection method that combines activation patterns from multiple layers of the detector's backbone and report its performance.