mlsanityData to 14 Oct 2026

Speechless: Speech Instruction Training Without Speech for Low Resource Languages

arXiv 2505.17417 ↗First posted 23 May 2025eess.AS cs.CL cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

The rapid growth of voice assistants powered by large language models (LLM) has highlighted a need for speech instruction data to train these systems. Despite the abundance of speech recognition data, there is a notable scarcity of speech instruction data, which is essential for fine-tuning models to understand and execute spoken commands. Generating high-quality synthetic speech requires a good text-to-speech (TTS) model, which may not be available to low resource languages. Our novel approach addresses this challenge by halting synthesis at the semantic representation level, bypassing the need for TTS. We achieve this by aligning synthetic semantic representations with the pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text instructions while maintaining the ability to understand spoken instructions during inference. This simplified training process is a promising approach to building voice assistant for low-resource languages.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

29 of 37 references.

References · 29

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  8. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  10. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
17 more
  1. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  2. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  3. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  4. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  6. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 79
  7. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  8. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 38
  9. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  10. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 31
  11. 2022Low-Resource Multilingual and Zero-Shot Multispeaker TTScited by 15
  12. 2024LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioningcited by 15
  13. 2024Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Modelscited by 10
  14. 2024Instruction Data Generation and Unsupervised Adaptation for Speech Language Modelscited by 9
  15. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  16. 2022An Analysis of Semantically-Aligned Speech-Text Embeddingscited by 5
  17. 2024ASTRA: Aligning Speech and Text Representations for Asr without Samplingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignmentcited by 1
  2. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASRcited by 1
  3. 2025Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Trackingcited by 1
  4. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1