# Learning test generators for cyber-physical systems

arXiv: https://arxiv.org/abs/2410.03202 · HTML: https://mlsanity.com/paper/2410.03202/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unrolled Generative Adversarial Networks](https://mlsanity.com/paper/1611.02163/) (2016)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search](https://mlsanity.com/paper/2107.06997/) (2021)
- [AmbieGen: A Search-based Framework for Autonomous Systems Testing](https://mlsanity.com/paper/2301.01234/) (2023)
- [Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification](https://mlsanity.com/paper/1910.02837/) (2019)
- [Falsification of Cyber-Physical Systems with Robustness-Guided Black-Box Checking](https://mlsanity.com/paper/2005.02126/) (2020)
- [Online GANs for Automatic Performance Testing](https://mlsanity.com/paper/2104.11069/) (2021)
- [Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation with Probabilistic Guarantees](https://mlsanity.com/paper/2110.10729/) (2021)
- [PSY-TaLiRo: A Python Toolbox for Search-Based Test Generation for Cyber-Physical Systems](https://mlsanity.com/paper/2106.02200/) (2021)
- [A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems](https://mlsanity.com/paper/2203.12138/) (2022)
- [Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions](https://mlsanity.com/paper/2207.11016/) (2022)
- [Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing](https://mlsanity.com/paper/2308.12762/) (2023)
- [Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical Systems](https://mlsanity.com/paper/2205.11060/) (2022)
- [Requirement falsification for cyber-physical systems using generative models](https://mlsanity.com/paper/2310.20493/) (2023)
- [Falsification of Multiple Requirements for Cyber-Physical Systems Using Online Generative Adversarial Networks and Multi-Armed Bandits](https://mlsanity.com/paper/2205.11057/) (2022)
- [WOGAN at the SBST 2022 CPS Tool Competition](https://mlsanity.com/paper/2205.11064/) (2022)
- [Diversity-guided Search Exploration for Self-driving Cars Test Generation through Frenet Space Encoding](https://mlsanity.com/paper/2401.14682/) (2024)
- [A Survey of the Metrics, Uses, and Subjects of Diversity-Based Techniques in Software Testing](https://mlsanity.com/paper/2311.09714/) (2023)

## Abstract (arXiv)

Black-box runtime verification methods for cyber-physical systems can be used to discover errors in systems whose inputs and outputs are expressed as signals over time and their correctness requirements are specified in a temporal logic. Existing methods, such as requirement falsification, often focus on finding a single input that is a counterexample to system correctness. In this paper, we study how to create test generators that can produce multiple and diverse counterexamples for a single requirement. Several counterexamples expose system failures in varying input conditions and support the root cause analysis of the faults. We present the WOGAN algorithm to create such test generators automatically. The algorithm works by training iteratively a Wasserstein generative adversarial network that models the target distribution of the uniform distribution on the set of counterexamples. WOGAN is an algorithm that trains generative models that act as test generators for runtime verification. The training is performed online without the need for a previous model or dataset. We also propose criteria to evaluate such test generators. We evaluate the trained generators on several well-known problems including the ARCH-COMP falsification benchmarks. Our experimental results indicate that generators trained by the WOGAN algorithm are as effective as state-of-the-art requirement falsification algorithms while producing tests that are as diverse as a sample from uniform random sampling. We conclude that WOGAN is a viable method to produce test generators automatically and that these test generators can generate multiple and diverse counterexamples for the runtime verification of cyber-physical systems.