# Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggering

arXiv: https://arxiv.org/abs/2008.02323 · HTML: https://mlsanity.com/paper/2008.02323/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2020.

## Built on by (most cited first, 8 of 8)

- [Keyword Transformer: A Self-Attention Model for Keyword Spotting](https://mlsanity.com/paper/2104.00769/) (2021)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [Streaming Transformer for Hardware Efficient Voice Trigger Detection and False Trigger Mitigation](https://mlsanity.com/paper/2105.06598/) (2021)
- [Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Models](https://mlsanity.com/paper/2203.15975/) (2022)
- [Streaming on-device detection of device directed speech from voice and touch-based invocation](https://mlsanity.com/paper/2110.04656/) (2021)
- [QbyE-MLPMixer: Query-by-Example Open-Vocabulary Keyword Spotting using MLPMixer](https://mlsanity.com/paper/2206.13231/) (2022)
- [Multi-task Learning with Cross Attention for Keyword Spotting](https://mlsanity.com/paper/2107.07634/) (2021)
- [Improving Voice Trigger Detection with Metric Learning](https://mlsanity.com/paper/2204.02455/) (2022)

## Abstract (arXiv)

We consider the design of two-pass voice trigger detection systems. We focus on the networks in the second pass that are used to re-score candidate segments obtained from the first-pass. Our baseline is an acoustic model(AM), with BiLSTM layers, trained by minimizing the CTC loss. We replace the BiLSTM layers with self-attention layers. Results on internal evaluation sets show that self-attention networks yield better accuracy while requiring fewer parameters. We add an auto-regressive decoder network on top of the self-attention layers and jointly minimize the CTC loss on the encoder and the cross-entropy loss on the decoder. This design yields further improvements over the baseline. We retrain all the models above in a multi-task learning(MTL) setting, where one branch of a shared network is trained as an AM, while the second branch classifies the whole sequence to be true-trigger or not. Results demonstrate that networks with self-attention layers yield ~60% relative reduction in false reject rates for a given false-alarm rate, while requiring 10% fewer parameters. When trained in the MTL setup, self-attention networks yield further accuracy improvements. On-device measurements show that we observe 70% relative reduction in inference time. Additionally, the proposed network architectures are ~5X faster to train.