# ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events

arXiv: https://arxiv.org/abs/2308.14550 · HTML: https://mlsanity.com/paper/2308.14550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method](https://mlsanity.com/paper/2003.01197/) (2020)
- [Adaptive Stress Testing for Autonomous Vehicles](https://mlsanity.com/paper/1902.01909/) (2019)
- [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018)
- [Adversarial Objects Against LiDAR-Based Autonomous Driving Systems](https://mlsanity.com/paper/1907.05418/) (2019)
- [Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving](https://mlsanity.com/paper/1903.10654/) (2019)
- [Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2006.13704/) (2020)
- [Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Driving](https://mlsanity.com/paper/2212.08787/) (2022)
- [Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation](https://mlsanity.com/paper/1908.01046/) (2019)
- [Attacking Vision-based Perception in End-to-End Autonomous Driving Models](https://mlsanity.com/paper/1910.01907/) (2019)
- [Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Driving](https://mlsanity.com/paper/1905.00229/) (2019)
- [Robustifying Reinforcement Learning Agents via Action Space Adversarial Training](https://mlsanity.com/paper/2007.07176/) (2020)
- [Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study](https://mlsanity.com/paper/1912.00805/) (2019)
- [Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles](https://mlsanity.com/paper/1812.06120/) (2018)
- [Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles](https://mlsanity.com/paper/2109.06126/) (2021)
- [DriveFuzz: Discovering Autonomous Driving Bugs through Driving Quality-Guided Fuzzing](https://mlsanity.com/paper/2211.01829/) (2022)
- [How Do We Fail? Stress Testing Perception in Autonomous Vehicles](https://mlsanity.com/paper/2203.14155/) (2022)
- [Query-based Targeted Action-Space Adversarial Policies on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2011.07114/) (2020)
- [Finding Physical Adversarial Examples for Autonomous Driving with Fast and Differentiable Image Compositing](https://mlsanity.com/paper/2010.08844/) (2020)

## Abstract (arXiv)

Autonomous vehicles are advanced driving systems that are well known to be vulnerable to various adversarial attacks, compromising vehicle safety and posing a risk to other road users. Rather than actively training complex adversaries by interacting with the environment, there is a need to first intelligently find and reduce the search space to only those states where autonomous vehicles are found to be less confident. In this paper, we propose a black-box testing framework ReMAV that uses offline trajectories first to analyze the existing behavior of autonomous vehicles and determine appropriate thresholds to find the probability of failure events. To this end, we introduce a three-step methodology which i) uses offline state action pairs of any autonomous vehicle under test, ii) builds an abstract behavior representation using our designed reward modeling technique to analyze states with uncertain driving decisions, and iii) uses a disturbance model for minimal perturbation attacks where the driving decisions are less confident. Our reward modeling technique helps in creating a behavior representation that allows us to highlight regions of likely uncertain behavior even when the standard autonomous vehicle performs well. We perform our experiments in a high-fidelity urban driving environment using three different driving scenarios containing single- and multi-agent interactions. Our experiment shows an increase in 35, 23, 48, and 50% in the occurrences of vehicle collision, road object collision, pedestrian collision, and offroad steering events, respectively by the autonomous vehicle under test, demonstrating a significant increase in failure events. We compare ReMAV with two baselines and show that ReMAV demonstrates significantly better effectiveness in generating failure events compared to the baselines in all evaluation metrics.