mlsanityData to 7 Oct 2026

Distilling an End-to-End Voice Assistant Without Instruction Training Data

arXiv 2410.02678 ↗First posted 3 Oct 2024cs.CL cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using >100x less training compute.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

47 of 60 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
35 more
  1. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  2. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  3. 2021AST: Audio Spectrogram Transformercited by 634
  4. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  5. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  8. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  10. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  11. 2023Listen, Think, and Understandcited by 206
  12. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  13. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  14. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  15. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  16. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  17. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  18. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  19. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  20. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  21. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  22. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 59
  23. 2023LLaSM: Large Language and Speech Modelcited by 56
  24. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  25. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  26. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 36
  27. 2021"How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversationscited by 30
  28. 2022STOP: A dataset for Spoken Task Oriented Semantic Parsingcited by 22
  29. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16
  30. 2023HeySQuAD: A Spoken Question Answering Datasetcited by 16
  31. 2023Svarah: Evaluating English ASR Systems on Indian Accentscited by 15
  32. 2023MD3: The Multi-Dialect Dataset of Dialoguescited by 12
  33. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  34. 2016Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phonescited by 9
  35. 2023Improving Speech Recognition for African American English With Audio Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 13.

35 papers.

Sort:

By area:

  1. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  3. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  4. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  5. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  6. 2025AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluationcited by 20
  7. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  8. 2025Closing the Gap Between Text and Speech Understanding in LLMscited by 14
  9. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  10. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  11. 2025Mind the Gap! Static and Interactive Evaluations of Large Audio Modelscited by 8
  12. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
23 more
  1. 2025ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systemscited by 7
  2. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  3. 2025DIFFA: Large Language Diffusion Models Can Listen and Understandcited by 7
  4. 2025SeaLLMs-Audio: Large Audio-Language Models for Southeast Asiacited by 7
  5. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  6. 2025DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilitiescited by 5
  7. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  8. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 5
  9. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  10. 2025ERIS: Evolutionary Real-world Interference Scheme for Jailbreaking Audio Large Modelscited by 2
  11. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understandingcited by 2
  12. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  13. 2025TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignmentcited by 1
  14. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  15. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRAcited by 1
  16. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  17. 2026The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfercited by 0
  18. 2026GRM: Utility-Aware Jailbreak Attacks on Audio LLMs via Gradient-Ratio Maskingcited by 0
  19. 2026Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Modelscited by 0
  20. 2026Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representationcited by 0
  21. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0
  22. 2026Audio Token Attention Is Predictable Before the Language Model Runscited by 0
  23. 2026High-Value Synthetic Supervision for Parameter-Efficient Adaptation of a Compact Japanese Speech Modelcited by 0