# A Scenario-Based Functional Testing Approach to Improving DNN Performance

arXiv: https://arxiv.org/abs/2307.07083 · HTML: https://mlsanity.com/paper/2307.07083/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 13 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [A Survey on Automated Driving System Testing: Landscapes and Trends](https://mlsanity.com/paper/2206.05961/) (2022)
- [Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology](https://mlsanity.com/paper/1905.03419/) (2019)
- [NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks](https://mlsanity.com/paper/2203.12915/) (2022)
- [Fundamental Considerations around Scenario-Based Testing for Automated Driving](https://mlsanity.com/paper/2005.04045/) (2020)
- [Testing Scenario Library Generation for Connected and Automated Vehicles, Part II: Case Studies](https://mlsanity.com/paper/1905.03428/) (2019)
- [Coverage based testing for V&V and Safety Assurance of Self-driving Autonomous Vehicles: A Systematic Literature Review](https://mlsanity.com/paper/2103.04364/) (2021)
- [The AI Driving Olympics at NeurIPS 2018](https://mlsanity.com/paper/1903.02503/) (2019)
- [Repairing DNN Architecture: Are We There Yet?](https://mlsanity.com/paper/2301.11568/) (2023)
- [Morphy: A Datamorphic Software Test Automation Tool](https://mlsanity.com/paper/1912.09881/) (2019)
- [Real-Time Optimal Trajectory Planning for Autonomous Vehicles and Lap Time Simulation Using Machine Learning](https://mlsanity.com/paper/2102.02315/) (2021)

## Built on by (most cited first, 7 of 7)

- [Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'](https://mlsanity.com/paper/2505.04950/) (2025)
- [User Centric Evaluation of Code Generation Tools](https://mlsanity.com/paper/2402.03130/) (2024)
- [Uncertainty-Aware Autonomous Vehicles: Predicting the Road Ahead](https://mlsanity.com/paper/2510.22680/) (2025)
- [ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation](https://mlsanity.com/paper/2406.12635/) (2024)
- [Robustness of Object Detection of Autonomous Vehicles in Adverse Weather Conditions](https://mlsanity.com/paper/2602.12902/) (2026)
- [FMCE-Net++: Feature Map Convergence Evaluation and Training](https://mlsanity.com/paper/2508.06109/) (2025)
- [Operational Robustness of LLMs on Code Generation](https://mlsanity.com/paper/2602.18800/) (2026)

## Abstract (arXiv)

This paper proposes a scenario-based functional testing approach for enhancing the performance of machine learning (ML) applications. The proposed method is an iterative process that starts with testing the ML model on various scenarios to identify areas of weakness. It follows by a further testing on the suspected weak scenarios and statistically evaluate the model's performance on the scenarios to confirm the diagnosis. Once the diagnosis of weak scenarios is confirmed by test results, the treatment of the model is performed by retraining the model using a transfer learning technique with the original model as the base and applying a set of training data specifically targeting the treated scenarios plus a subset of training data selected at random from the original train dataset to prevent the so-call catastrophic forgetting effect. Finally, after the treatment, the model is assessed and evaluated again by testing on the treated scenarios as well as other scenarios to check if the treatment is effective and no side effect caused. The paper reports a case study with a real ML deep neural network (DNN) model, which is the perception system of an autonomous racing car. It is demonstrated that the method is effective in the sense that DNN model's performance can be improved. It provides an efficient method of enhancing ML model's performance with much less human and compute resource than retrain from scratch.