# GE2E-KWS: Generalized End-to-End Training and Evaluation for Zero-shot Keyword Spotting

arXiv: https://arxiv.org/abs/2410.16647 · HTML: https://mlsanity.com/paper/2410.16647/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [Streaming keyword spotting on mobile devices](https://mlsanity.com/paper/2005.06720/) (2020)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [End-to-End Streaming Keyword Spotting](https://mlsanity.com/paper/1812.02802/) (2018)
- [Training Keyword Spotters with Limited and Synthesized Speech Data](https://mlsanity.com/paper/2002.01322/) (2020)
- [TristouNet: Triplet Loss for Speaker Turn Embedding](https://mlsanity.com/paper/1609.04301/) (2016)
- [Extending Multilingual Speech Synthesis to 100+ Languages without Transcribed Data](https://mlsanity.com/paper/2402.18932/) (2024)
- [Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Models](https://mlsanity.com/paper/2402.17184/) (2024)
- [Triplet Network with Attention for Speaker Diarization](https://mlsanity.com/paper/1808.01535/) (2018)
- [EfficientWord-Net: An Open Source Hotword Detection Engine based on One-shot Learning](https://mlsanity.com/paper/2111.00379/) (2021)
- [Locale Encoding For Scalable Multilingual Keyword Spotting Models](https://mlsanity.com/paper/2302.12961/) (2023)

## Built on by (most cited first, 2 of 2)

- [GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examples](https://mlsanity.com/paper/2505.14814/) (2025)
- [Zero-Shot KWS for Children's Speech using Layer-Wise Features from SSL Models](https://mlsanity.com/paper/2508.21248/) (2025)

## Abstract (arXiv)

We propose GE2E-KWS -- a generalized end-to-end training and evaluation framework for customized keyword spotting. Specifically, enrollment utterances are separated and grouped by keywords from the training batch and their embedding centroids are compared to all other test utterance embeddings to compute the loss. This simulates runtime enrollment and verification stages, and improves convergence stability and training speed by optimizing matrix operations compared to SOTA triplet loss approaches. To benchmark different models reliably, we propose an evaluation process that mimics the production environment and compute metrics that directly measure keyword matching accuracy. Trained with GE2E loss, our 419KB quantized conformer model beats a 7.5GB ASR encoder by 23.6% relative AUC, and beats a same size triplet loss model by 60.7% AUC. Our KWS models are natively streamable with low memory footprints, and designed to continuously run on-device with no retraining needed for new keywords (zero-shot).