# AB/BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy

arXiv: https://arxiv.org/abs/2204.08474 · HTML: https://mlsanity.com/paper/2204.08474/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 6 of 6)

- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Large-Scale Domain Adaptation via Teacher-Student Learning](https://mlsanity.com/paper/1708.05466/) (2017)
- [Attention-based End-to-End Models for Small-Footprint Keyword Spotting](https://mlsanity.com/paper/1803.10916/) (2018)
- [Data Augmentation for Robust Keyword Spotting under Playback Interference](https://mlsanity.com/paper/1808.00563/) (2018)
- [On Front-end Gain Invariant Modeling for Wake Word Spotting](https://mlsanity.com/paper/2010.06676/) (2020)

## Abstract (arXiv)

Evaluation of keyword spotting (KWS) systems that detect keywords in speech is a challenging task under realistic privacy constraints. The KWS is designed to only collect data when the keyword is present, limiting the availability of hard samples that may contain false negatives, and preventing direct estimation of model recall from production data. Alternatively, complementary data collected from other sources may not be fully representative of the real application. In this work, we propose an evaluation technique which we call AB/BA analysis. Our framework evaluates a candidate KWS model B against a baseline model A, using cross-dataset offline decoding for relative recall estimation, without requiring negative examples. Moreover, we propose a formulation with assumptions that allow estimation of relative false positive rate between models with low variance even when the number of false positives is small. Finally, we propose to leverage machine-generated soft labels, in a technique we call Semi-Supervised AB/BA analysis, that improves the analysis time, privacy, and cost. Experiments with both simulation and real data show that AB/BA analysis is successful at measuring recall improvement in conjunction with the trade-off in relative false positive rate.