mlsanityData to 7 Oct 2026

SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions

arXiv 2501.19377 ↗First posted 31 Jan 2025cs.SD cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

28 of 39 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  9. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  12. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
16 more
  1. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  2. 2023Speed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizationscited by 45
  3. 2020Accurate Detection of Wake Word Start and End Using a CNNcited by 24
  4. 2018Device-directed Utterance Detectioncited by 19
  5. 2019Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directedcited by 10
  6. 2021Streaming Transformer for Hardware Efficient Voice Trigger Detection and False Trigger Mitigationcited by 8
  7. 2020Lattice-based Improvements for Voice Triggering Using Graph Neural Networkscited by 7
  8. 2020Voice trigger detection from LVCSR hypothesis lattices using bidirectional lattice recurrent neural networkscited by 7
  9. 2024Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detectioncited by 6
  10. 2022Device-Directed Speech Detection: Regularization via Distillation for Weakly-Supervised Modelscited by 5
  11. 2023Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Modelscited by 5
  12. 2024A Multimodal Approach to Device-Directed Speech Detection with Large Language Modelscited by 5
  13. 2023Contrastive Speech Mixup for Low-resource Keyword Spottingcited by 4
  14. 2023Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Featurescited by 4
  15. 2022Low-resource Low-footprint Wake-word Detection using Knowledge Distillationcited by 3
  16. 2023Multichannel Voice Trigger Detection Based on Transform-average-concatenatecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
  2. 2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safetycited by 7
  3. 2025Adaptive Knowledge Distillation for Device-Directed Speech Detectioncited by 2
  4. 2025Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretrainingcited by 0
  5. 2026Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AIcited by 0
  6. 2026AVA-VLM: Adaptive Visual Attention-Vision Language Model for In-the-Wild Construction Site Monitoringcited by 0