# Coverage Guided Testing for Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1911.01952 · HTML: https://mlsanity.com/paper/1911.01952/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2019.

## Built on by (most cited first, 15 of 15)

- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [MTTM: Metamorphic Testing for Textual Content Moderation Software](https://mlsanity.com/paper/2302.05706/) (2023)
- [Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance](https://mlsanity.com/paper/2112.00646/) (2021)
- [AEON: A Method for Automatic Evaluation of NLP Test Cases](https://mlsanity.com/paper/2205.06439/) (2022)
- [PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing](https://mlsanity.com/paper/2111.10481/) (2021)
- [Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles](https://mlsanity.com/paper/2106.01258/) (2021)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications](https://mlsanity.com/paper/2108.10451/) (2021)
- [Hierarchical Testing with Rabbit Optimization for Industrial Cyber-Physical Systems](https://mlsanity.com/paper/2507.04100/) (2025)
- [Short-term Load Forecasting with Distributed Long Short-Term Memory](https://mlsanity.com/paper/2208.01147/) (2022)
- [Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features](https://mlsanity.com/paper/2103.03704/) (2021)
- [DREAM: Debugging and Repairing AutoML Pipelines](https://mlsanity.com/paper/2401.00379/) (2023)
- [LeCov: Multi-level Testing Criteria for Large Language Models](https://mlsanity.com/paper/2408.10474/) (2024)

## Abstract (arXiv)

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this paper aims to develop a coverage guided testing approach to systematically exploit the internal behaviour of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both step-wise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool TestRNN, which is then evaluated on a set of LSTM benchmarks. Experiments confirm that TestRNN has advantages over the state-of-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, TestRNN enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step towards interpretable neural network testing.