# Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems

arXiv: https://arxiv.org/abs/2402.18393 · HTML: https://mlsanity.com/paper/2402.18393/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 81 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving](https://mlsanity.com/paper/1907.06826/) (2019)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Globally-Robust Neural Networks](https://mlsanity.com/paper/2102.08452/) (2021)
- [On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2201.05057/) (2022)
- [AdvDO: Realistic Adversarial Attacks for Trajectory Prediction](https://mlsanity.com/paper/2209.08744/) (2022)
- [Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles](https://mlsanity.com/paper/2104.07830/) (2021)
- [You Can't See Me: Physical Removal Attacks on LiDAR-based Autonomous Vehicles Driving Frameworks](https://mlsanity.com/paper/2210.09482/) (2022)
- [Fooling LiDAR Perception via Adversarial Trajectory Perturbation](https://mlsanity.com/paper/2103.15326/) (2021)
- [Distribution-Aware Testing of Neural Networks Using Generative Models](https://mlsanity.com/paper/2102.13602/) (2021)
- [Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?](https://mlsanity.com/paper/2106.12233/) (2021)
- [Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles](https://mlsanity.com/paper/2109.06126/) (2021)
- [Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems](https://mlsanity.com/paper/2210.15432/) (2022)
- [DriveFuzz: Discovering Autonomous Driving Bugs through Driving Quality-Guided Fuzzing](https://mlsanity.com/paper/2211.01829/) (2022)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems](https://mlsanity.com/paper/2307.07493/) (2023)
- [DeepCrashTest: Turning Dashcam Videos into Virtual Crash Tests for Automated Driving Systems](https://mlsanity.com/paper/2003.11766/) (2020)
- [Optimal Vehicle Path Planning Using Quadratic Optimization for Baidu Apollo Open Platform](https://mlsanity.com/paper/2112.02132/) (2021)
- [Scenario-Based Test Reduction and Prioritization for Multi-Module Autonomous Driving Systems](https://mlsanity.com/paper/2209.01546/) (2022)
- [MultiTest: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems](https://mlsanity.com/paper/2401.14314/) (2024)
- [Benchmarking Robustness of AI-Enabled Multi-sensor Fusion Systems: Challenges and Opportunities](https://mlsanity.com/paper/2306.03454/) (2023)

## Built on by (most cited first, 4 of 4)

- [DriveTester: A Unified Platform for Simulation-Based Autonomous Driving Testing](https://mlsanity.com/paper/2412.12656/) (2024)
- [From Particles to Perils: SVGD-Based Hazardous Scenario Generation for Autonomous Driving Systems Testing](https://mlsanity.com/paper/2604.18918/) (2026)
- [ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2605.29114/) (2026)
- [Learning from the Test: Self-Referential Differential Testing for Deep RL Agents](https://mlsanity.com/paper/2608.22284/) (2026)

## Abstract (arXiv)

Autonomous Driving System (ADS) testing is crucial in ADS development, with the current primary focus being on safety. However, the evaluation of non-safety-critical performance, particularly the ADS's ability to make optimal decisions and produce optimal paths for autonomous vehicles (AVs), is also vital to ensure the intelligence and reduce risks of AVs. Currently, there is little work dedicated to assessing the robustness of ADSs' path-planning decisions (PPDs), i.e., whether an ADS can maintain the optimal PPD after an insignificant change in the environment. The key challenges include the lack of clear oracles for assessing PPD optimality and the difficulty in searching for scenarios that lead to non-optimal PPDs. To fill this gap, in this paper, we focus on evaluating the robustness of ADSs' PPDs and propose the first method, Decictor, for generating non-optimal decision scenarios (NoDSs), where the ADS does not plan optimal paths for AVs. Decictor comprises three main components: Non-invasive Mutation, Consistency Check, and Feedback. To overcome the oracle challenge, Non-invasive Mutation is devised to implement conservative modifications, ensuring the preservation of the original optimal path in the mutated scenarios. Subsequently, the Consistency Check is applied to determine the presence of non-optimal PPDs by comparing the driving paths in the original and mutated scenarios. To deal with the challenge of large environment space, we design Feedback metrics that integrate spatial and temporal dimensions of the AV's movement. These metrics are crucial for effectively steering the generation of NoDSs. We evaluate Decictor on Baidu Apollo, an open-source and production-grade ADS. The experimental results validate the effectiveness of Decictor in detecting non-optimal PPDs of ADSs.