# Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems

arXiv: https://arxiv.org/abs/2210.15432 · HTML: https://mlsanity.com/paper/2210.15432/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 43 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [Adaptive Stress Testing for Autonomous Vehicles](https://mlsanity.com/paper/1902.01909/) (2019)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [Adaptive Stress Testing with Reward Augmentation for Autonomous Vehicle Validation](https://mlsanity.com/paper/1908.01046/) (2019)
- [Can Offline Testing of Deep Neural Networks Replace Their Online Testing?](https://mlsanity.com/paper/2101.11118/) (2021)
- [Automatic Web Testing using Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2103.06018/) (2021)

## Built on by (most cited first, 19 of 19)

- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Boundary State Generation for Testing and Improvement of Autonomous Driving Systems](https://mlsanity.com/paper/2307.10590/) (2023)
- [EpiTESTER: Testing Autonomous Vehicles with Epigenetic Algorithm and Attention Mechanism](https://mlsanity.com/paper/2312.00207/) (2023)
- [Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approach](https://mlsanity.com/paper/2311.07377/) (2023)
- [Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems](https://mlsanity.com/paper/2311.18768/) (2023)
- [Decictor: Towards Evaluating the Robustness of Decision-Making in Autonomous Driving Systems](https://mlsanity.com/paper/2402.18393/) (2024)
- [DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data](https://mlsanity.com/paper/2310.05170/) (2023)
- [Reinforcement Learning for Online Testing of Autonomous Driving Systems: a Replication and Extension Study](https://mlsanity.com/paper/2403.13729/) (2024)
- [ADReFT: Adaptive Decision Repair for Safe Autonomous Driving via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.23960/) (2025)
- [Advancing Autonomous Driving System Testing: Demands, Challenges, and Future Directions](https://mlsanity.com/paper/2512.11887/) (2025)
- [Natural Adversaries: Fuzzing Autonomous Vehicles with Realistic Roadside Object Placements](https://mlsanity.com/paper/2409.10562/) (2024)
- [MARL-OT: Multi-Agent Reinforcement Learning Guided Online Fuzzing to Detect Safety Violation in Autonomous Driving Systems](https://mlsanity.com/paper/2501.14451/) (2025)
- [GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth](https://mlsanity.com/paper/2503.15953/) (2025)
- [STCLocker: Deadlock Avoidance Testing for Autonomous Driving Systems](https://mlsanity.com/paper/2506.23995/) (2025)
- [Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents](https://mlsanity.com/paper/2604.25161/) (2026)
- [Gimitest: A Comprehensive Tool for Testing Reinforcement Learning Policies](https://mlsanity.com/paper/2607.07029/) (2026)
- [Evaluating Fuzz Testing for Reinforcement Learning Agents](https://mlsanity.com/paper/2607.24577/) (2026)
- [Learning from the Test: Self-Referential Differential Testing for Deep RL Agents](https://mlsanity.com/paper/2608.22284/) (2026)
- [DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2610.04494/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) have been widely used to perform real-world tasks in cyber-physical systems such as Autonomous Driving Systems (ADS). Ensuring the correct behavior of such DNN-Enabled Systems (DES) is a crucial topic. Online testing is one of the promising modes for testing such systems with their application environments (simulated or real) in a closed loop taking into account the continuous interaction between the systems and their environments. However, the environmental variables (e.g., lighting conditions) that might change during the systems' operation in the real world, causing the DES to violate requirements (safety, functional), are often kept constant during the execution of an online test scenario due to the two major challenges: (1) the space of all possible scenarios to explore would become even larger if they changed and (2) there are typically many requirements to test simultaneously. In this paper, we present MORLOT (Many-Objective Reinforcement Learning for Online Testing), a novel online testing approach to address these challenges by combining Reinforcement Learning (RL) and many-objective search. MORLOT leverages RL to incrementally generate sequences of environmental changes while relying on many-objective search to determine the changes so that they are more likely to achieve any of the uncovered objectives. We empirically evaluate MORLOT using CARLA, a high-fidelity simulator widely used for autonomous driving research, integrated with Transfuser, a DNN-enabled ADS for end-to-end driving. The evaluation results show that MORLOT is significantly more effective and efficient than alternatives with a large effect size. In other words, MORLOT is a good option to test DES with dynamically changing environments while accounting for multiple safety requirements.