# Streaming on-device detection of device directed speech from voice and touch-based invocation

arXiv: https://arxiv.org/abs/2110.04656 · HTML: https://mlsanity.com/paper/2110.04656/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [1D Convolutional Neural Networks and Applications: A Survey](https://mlsanity.com/paper/1905.03554/) (2019)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [Transformer ASR with Contextual Block Processing](https://mlsanity.com/paper/1910.07204/) (2019)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning](https://mlsanity.com/paper/2109.07327/) (2021)
- [Wake Word Detection with Streaming Transformers](https://mlsanity.com/paper/2102.04488/) (2021)
- [Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggering](https://mlsanity.com/paper/2008.02323/) (2020)
- [Streaming Transformer for Hardware Efficient Voice Trigger Detection and False Trigger Mitigation](https://mlsanity.com/paper/2105.06598/) (2021)
- [Progressive Voice Trigger Detection: Accuracy vs Latency](https://mlsanity.com/paper/2010.15446/) (2020)
- [Multi-task Learning for Voice Trigger Detection](https://mlsanity.com/paper/2001.09519/) (2020)
- [Knowledge Transfer for Efficient On-device False Trigger Mitigation](https://mlsanity.com/paper/2010.10591/) (2020)
- [Streaming ResLSTM with Causal Mean Aggregation for Device-Directed Utterance Detection](https://mlsanity.com/paper/2007.09245/) (2020)

## Built on by (most cited first, 4 of 4)

- [Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection](https://mlsanity.com/paper/2406.09617/) (2024)
- [Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models](https://mlsanity.com/paper/2312.03632/) (2023)
- [Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Features](https://mlsanity.com/paper/2310.15261/) (2023)
- [Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI](https://mlsanity.com/paper/2604.08412/) (2026)

## Abstract (arXiv)

When interacting with smart devices such as mobile phones or wearables, the user typically invokes a virtual assistant (VA) by saying a keyword or by pressing a button on the device. However, in many cases, the VA can accidentally be invoked by the keyword-like speech or accidental button press, which may have implications on user experience and privacy. To this end, we propose an acoustic false-trigger-mitigation (FTM) approach for on-device device-directed speech detection that simultaneously handles the voice-trigger and touch-based invocation. To facilitate the model deployment on-device, we introduce a new streaming decision layer, derived using the notion of temporal convolutional networks (TCN) [1], known for their computational efficiency. To the best of our knowledge, this is the first approach that can detect device-directed speech from more than one invocation type in a streaming fashion. We compare this approach with streaming alternatives based on vanilla Average layer, and canonical LSTMs, and show: (i) that all the models show only a small degradation in accuracy compared with the invocation-specific models, and (ii) that the newly introduced streaming TCN consistently performs better or comparable with the alternatives, while mitigating device undirected speech faster in time, and with (relative) reduction in runtime peak-memory over the LSTM-based approach of 33% vs. 7%, when compared to a non-streaming counterpart.