# CrossASR++: A Modular Differential Testing Framework for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2105.14881 · HTML: https://mlsanity.com/paper/2105.14881/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2021.

## Built on by (most cited first, 4 of 4)

- [Stealthy Backdoor Attack for Code Models](https://mlsanity.com/paper/2301.02496/) (2023)
- [BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets](https://mlsanity.com/paper/2210.04688/) (2022)
- [Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking](https://mlsanity.com/paper/2302.08018/) (2023)
- [ASDF: A Differential Testing Framework for Automatic Speech Recognition Systems](https://mlsanity.com/paper/2302.05582/) (2023)

## Abstract (arXiv)

Developers need to perform adequate testing to ensure the quality of Automatic Speech Recognition (ASR) systems. However, manually collecting required test cases is tedious and time-consuming. Our recent work proposes CrossASR, a differential testing method for ASR systems. This method first utilizes Text-to-Speech (TTS) to generate audios from texts automatically and then feed these audios into different ASR systems for cross-referencing to uncover failed test cases. It also leverages a failure estimator to find failing test cases more efficiently. Such a method is inherently self-improvable: the performance can increase by leveraging more advanced TTS and ASR systems. So in this accompanying tool demo paper, we devote more engineering and propose CrossASR++, an easy-to-use ASR testing tool that can be conveniently extended to incorporate different TTS and ASR systems, and failure estimators. We also make CrossASR++ chunk texts from a given corpus dynamically and enable the estimator to work in a more effective and flexible way. We demonstrate that the new features can help CrossASR++ discover more failed test cases. Using the same TTS and ASR systems, CrossASR++ can uncover 26.2% more failed test cases for 4 ASRs than the original tool. Moreover, by simply adding one more ASR for cross-referencing, we can increase the number of failed test cases uncovered for each of the 4 ASR systems by 25.07%, 39.63%, 20.9% and 8.17% respectively. We also extend CrossASR++ with 5 additional failure estimators. Compared to worst estimator, the best one can discover 10.41% more failed test cases within the same amount of time.