mlsanityData to 7 Oct 2026

DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment

arXiv 2406.18871 ↗First posted 27 Jun 2024eess.AS cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities, enabling SLMs to interpret and generate comprehensive natural language descriptions, thereby facilitating the capability to understand both linguistic and non-linguistic features in speech. Enhanced with the proposed approach, our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. Moreover, we discover that the aligned model exhibits a zero-shot instruction-following capability without explicit speech instruction tuning. These findings highlight the potential to reshape instruction-following SLMs by incorporating rich, descriptive speech captions.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

29 of 31 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
17 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  3. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  4. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  5. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  6. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  7. 2023Listen, Think, and Understandcited by 206
  8. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  9. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  10. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  11. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
  12. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  13. 2023Joint Audio and Speech Understandingcited by 99
  14. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  15. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  16. 2023LLaSM: Large Language and Speech Modelcited by 56
  17. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 18; 2026: 8.

32 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  3. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  4. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  5. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  6. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  7. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  8. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  9. 2025Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decodingcited by 22
  10. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  11. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  12. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
20 more
  1. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  2. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  3. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
  4. 2024SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioningcited by 6
  5. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  6. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 5
  7. 2025Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Modelscited by 4
  8. 2025LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMscited by 3
  9. 2025Towards Reliable Large Audio Language Modelcited by 3
  10. 2025AHAMask: Reliable Task Specification for Large Audio Language Models without Instructionscited by 3
  11. 2025Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoningcited by 3
  12. 2025ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Modelscited by 3
  13. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
  14. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
  15. 2026Investigating the Viability of Employing Multi-modal Large Language Models in the Context of Audio Deepfake Detectioncited by 1
  16. 2026ALARM: Audio-Language Alignment for Reasoning Modelscited by 1
  17. 2025MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languagescited by 0
  18. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Modelscited by 0
  19. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0
  20. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models
  2. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning
  3. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
  4. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation
  5. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs
  6. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs
  7. 2025Enhancing Speech Large Language Models through Reinforced Behavior Alignment
  8. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment