mlsanityData to 7 Oct 2026

Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech

arXiv 2410.01162 ↗First posted 2 Oct 2024eess.AS cs.CL cs.SDcited by 18

Not on the map. Why?

Abstract (arXiv)

This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt that has also been conditioned on the user's speaking style. This framework enables the encoder to generate tokens that capture both linguistic and paralinguistic information and effectively convey them to the LLM, even when the LLM's weights remain completely frozen. To the best of our knowledge, our work is the first to explore how to induce a frozen LLM to understand more than just linguistic content from speech inputs in a general interaction setting. Experiments demonstrate that our system is able to produce higher quality and more empathetic responses to expressive speech prompts compared to several baselines.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

30 of 34 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
18 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2023Listen, Think, and Understandcited by 206
  3. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  4. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  5. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  6. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  7. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  8. 2023Joint Audio and Speech Understandingcited by 99
  9. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  10. 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 75
  11. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  12. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  13. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  14. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  15. 2023SECap: Speech Emotion Captioning with Large Language Modelcited by 33
  16. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  17. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 8
  18. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 8.

18 papers.

Sort:

By area:

  1. 2025Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild Datacited by 10
  2. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  3. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
  4. 2025Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMscited by 6
  5. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  6. 2026Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Modelscited by 5
  7. 2025Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generationcited by 4
  8. 2025Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Modelscited by 4
  9. 2025Towards Reliable Large Audio Language Modelcited by 3
  10. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuningcited by 2
  11. 2025Dual Information Speech Language Models for Emotional Conversationscited by 0
  12. 2025MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languagescited by 0
6 more
  1. 2026RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuancecited by 0
  2. 2026SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Modelscited by 0
  3. 2026Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognitioncited by 0
  4. 2026CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agentscited by 0
  5. 2026Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuningcited by 0
  6. 2026Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMscited by 0