mlsanityData to 7 Oct 2026

A Multimodal Approach to Device-Directed Speech Detection with Large Language Models

arXiv 2403.14438 ↗First posted 21 Mar 2024cs.CL cs.LG eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Interactions with virtual assistants typically start with a predefined trigger phrase followed by the user command. To make interactions with the assistant more intuitive, we explore whether it is feasible to drop the requirement that users must begin each command with a trigger phrase. We explore this task in three ways: First, we train classifiers using only acoustic information obtained from the audio waveform. Second, we take the decoder outputs of an automatic speech recognition (ASR) system, such as 1-best hypotheses, as input features to a large language model (LLM). Finally, we explore a multimodal system that combines acoustic and lexical features, as well as ASR decoder signals in an LLM. Using multimodal information yields relative equal-error-rate improvements over text-only and audio-only models of up to 39% and 61%. Increasing the size of the LLM and training with low-rank adaption leads to further relative EER reductions of up to 18% on our dataset.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

34 of 50 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
22 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  5. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  6. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  7. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  8. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  9. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  10. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  11. 2023Prefix tuning for automated audio captioningcited by 26
  12. 2020Accurate Detection of Wake Word Start and End Using a CNNcited by 24
  13. 2018Device-directed Utterance Detectioncited by 19
  14. 2020Multi-task Learning for Speaker Verification and Voice Trigger Detectioncited by 9
  15. 2023Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognitioncited by 8
  16. 2020Improving Device Directedness Classification of Utterances with Semantic Lexical Featurescited by 6
  17. 2022Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Modelscited by 5
  18. 2023Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Modelscited by 5
  19. 2022Audio-to-Intent Using Acoustic-Textual Subword Representations from End-to-End ASRcited by 4
  20. 2023Contrastive Speech Mixup for Low-resource Keyword Spottingcited by 4
  21. 2022Low-resource Low-footprint Wake-word Detection using Knowledge Distillationcited by 3
  22. 2023Efficient Multimodal Neural Networks for Trigger-less Voice Assistantscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  2. 2024Large Language Models for Dysfluency Detection in Stuttered Speechcited by 8
  3. 2024LLMs are One-Shot URL Classifiers and Explainerscited by 6
  4. 2025SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactionscited by 6
  5. 2026Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AIcited by 0