# Streaming Transformer for Hardware Efficient Voice Trigger Detection and False Trigger Mitigation

arXiv: https://arxiv.org/abs/2105.06598 · HTML: https://mlsanity.com/paper/2105.06598/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Transformer ASR with Contextual Block Processing](https://mlsanity.com/paper/1910.07204/) (2019)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [A Cascade Architecture for Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1712.03603/) (2017)
- [Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed](https://mlsanity.com/paper/1902.00570/) (2019)
- [Wake Word Detection with Streaming Transformers](https://mlsanity.com/paper/2102.04488/) (2021)
- [Optimize what matters: Training DNN-HMM Keyword Spotting Model Using End Metric](https://mlsanity.com/paper/2011.01151/) (2020)
- [Unacceptable, where is my privacy? Exploring Accidental Triggers of Smart Speakers](https://mlsanity.com/paper/2008.00508/) (2020)
- [Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggering](https://mlsanity.com/paper/2008.02323/) (2020)
- [Lattice-based Improvements for Voice Triggering Using Graph Neural Networks](https://mlsanity.com/paper/2001.10822/) (2020)
- [Voice trigger detection from LVCSR hypothesis lattices using bidirectional lattice recurrent neural networks](https://mlsanity.com/paper/2003.00304/) (2020)
- [Improving Device Directedness Classification of Utterances with Semantic Lexical Features](https://mlsanity.com/paper/2010.01949/) (2020)
- [Progressive Voice Trigger Detection: Accuracy vs Latency](https://mlsanity.com/paper/2010.15446/) (2020)
- [Multi-task Learning for Voice Trigger Detection](https://mlsanity.com/paper/2001.09519/) (2020)
- [Knowledge Transfer for Efficient On-device False Trigger Mitigation](https://mlsanity.com/paper/2010.10591/) (2020)
- [Complementary Language Model and Parallel Bi-LRNN for False Trigger Mitigation](https://mlsanity.com/paper/2008.08113/) (2020)
- [Streaming ResLSTM with Causal Mean Aggregation for Device-Directed Utterance Detection](https://mlsanity.com/paper/2007.09245/) (2020)

## Built on by (most cited first, 8 of 8)

- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions](https://mlsanity.com/paper/2501.19377/) (2025)
- [Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Models](https://mlsanity.com/paper/2203.15975/) (2022)
- [Audiomer: A Convolutional Transformer For Keyword Spotting](https://mlsanity.com/paper/2109.10252/) (2021)
- [Streaming on-device detection of device directed speech from voice and touch-based invocation](https://mlsanity.com/paper/2110.04656/) (2021)
- [Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Features](https://mlsanity.com/paper/2310.15261/) (2023)
- [To Wake-up or Not to Wake-up: Reducing Keyword False Alarm by Successive Refinement](https://mlsanity.com/paper/2304.03416/) (2023)
- [Streaming Anchor Loss: Augmenting Supervision with Temporal Significance](https://mlsanity.com/paper/2310.05886/) (2023)

## Abstract (arXiv)

We present a unified and hardware efficient architecture for two stage voice trigger detection (VTD) and false trigger mitigation (FTM) tasks. Two stage VTD systems of voice assistants can get falsely activated to audio segments acoustically similar to the trigger phrase of interest. FTM systems cancel such activations by using post trigger audio context. Traditional FTM systems rely on automatic speech recognition lattices which are computationally expensive to obtain on device. We propose a streaming transformer (TF) encoder architecture, which progressively processes incoming audio chunks and maintains audio context to perform both VTD and FTM tasks using only acoustic features. The proposed joint model yields an average 18% relative reduction in false reject rate (FRR) for the VTD task at a given false alarm rate. Moreover, our model suppresses 95% of the false triggers with an additional one second of post-trigger audio. Finally, on-device measurements show 32% reduction in runtime memory and 56% reduction in inference time compared to non-streaming version of the model.