# Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models

arXiv: https://arxiv.org/abs/2312.03632 · HTML: https://mlsanity.com/paper/2312.03632/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Prefix tuning for automated audio captioning](https://mlsanity.com/paper/2303.17489/) (2023)
- [Accurate Detection of Wake Word Start and End Using a CNN](https://mlsanity.com/paper/2008.03790/) (2020)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [Multi-task Learning for Speaker Verification and Voice Trigger Detection](https://mlsanity.com/paper/2001.10816/) (2020)
- [Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition](https://mlsanity.com/paper/2304.08862/) (2023)
- [Improving Device Directedness Classification of Utterances with Semantic Lexical Features](https://mlsanity.com/paper/2010.01949/) (2020)
- [Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Models](https://mlsanity.com/paper/2203.15975/) (2022)
- [Streaming on-device detection of device directed speech from voice and touch-based invocation](https://mlsanity.com/paper/2110.04656/) (2021)
- [Audio-to-Intent Using Acoustic-Textual Subword Representations from End-to-End ASR](https://mlsanity.com/paper/2210.12134/) (2022)
- [Contrastive Speech Mixup for Low-resource Keyword Spotting](https://mlsanity.com/paper/2305.01170/) (2023)
- [Low-resource Low-footprint Wake-word Detection using Knowledge Distillation](https://mlsanity.com/paper/2207.03331/) (2022)
- [Implicit Acoustic Echo Cancellation for Keyword Spotting and Device-Directed Speech Detection](https://mlsanity.com/paper/2111.10639/) (2021)

## Built on by (most cited first, 5 of 5)

- [SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions](https://mlsanity.com/paper/2501.19377/) (2025)
- [Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection](https://mlsanity.com/paper/2406.09617/) (2024)
- [A Multimodal Approach to Device-Directed Speech Detection with Large Language Models](https://mlsanity.com/paper/2403.14438/) (2024)
- [Device-Directed Speech Detection for Follow-up Conversations Using Large Language Models](https://mlsanity.com/paper/2411.00023/) (2024)
- [Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI](https://mlsanity.com/paper/2604.08412/) (2026)

## Abstract (arXiv)

Interactions with virtual assistants typically start with a trigger phrase followed by a command. In this work, we explore the possibility of making these interactions more natural by eliminating the need for a trigger phrase. Our goal is to determine whether a user addressed the virtual assistant based on signals obtained from the streaming audio recorded by the device microphone. We address this task by combining 1-best hypotheses and decoder signals from an automatic speech recognition system with acoustic representations from an audio encoder as input features to a large language model (LLM). In particular, we are interested in data and resource efficient systems that require only a small amount of training data and can operate in scenarios with only a single frozen LLM available on a device. For this reason, our model is trained on 80k or less examples of multimodal data using a combination of low-rank adaptation and prefix tuning. We compare the proposed system to unimodal baselines and show that the multimodal approach achieves lower equal-error-rates (EERs), while using only a fraction of the training data. We also show that low-dimensional specialized audio representations lead to lower EERs than high-dimensional general audio representations.