mlsanityData to 14 Oct 2026

BLSP-Emo: Towards Empathetic Large Speech-Language Models

arXiv 2406.03872 ↗First posted 6 Jun 2024cs.CL cs.SD eess.AScited by 36

Not on the map. Why?

Abstract (arXiv)

The recent release of GPT-4o showcased the potential of end-to-end multimodal models, not just in terms of low latency but also in their ability to understand and generate expressive speech with rich emotions. While the details are unknown to the open research community, it likely involves significant amounts of curated data and compute, neither of which is readily accessible. In this paper, we present BLSP-Emo (Bootstrapped Language-Speech Pretraining with Emotion support), a novel approach to developing an end-to-end speech-language model capable of understanding both semantics and emotions in speech and generate empathetic responses. BLSP-Emo utilizes existing speech recognition (ASR) and speech emotion recognition (SER) datasets through a two-stage process. The first stage focuses on semantic alignment, following recent work on pretraining speech-language models using ASR data. The second stage performs emotion alignment with the pretrained speech-language model on an emotion-aware continuation task constructed from SER data. Our experiments demonstrate that the BLSP-Emo model excels in comprehending speech and delivering empathetic responses, both in instruction-following tasks and conversations.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 18 of its 40 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

18 of 40 references.

References · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  10. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  12. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
6 more
  1. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  2. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  3. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  4. 2023LLaSM: Large Language and Speech Modelcited by 58
  5. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  6. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 18.

36 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  3. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 31
  4. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  5. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 25
  6. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  7. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  8. 2025Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samplescited by 11
  9. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  10. 2026All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluationcited by 8
  11. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Datacited by 7
  12. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Datacited by 7
24 more
  1. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  2. 2025Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generationcited by 4
  3. 2025Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Modelscited by 4
  4. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  5. 2025Empathy Omni: Enabling Empathetic Speech Response Generation through Large Language Modelscited by 3
  6. 2026Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialoguecited by 2
  7. 2026Resurfacing Paralinguistic Awareness in Large Audio Language Modelscited by 2
  8. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  9. 2026Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioningcited by 1
  10. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  11. 2025Voice Interaction With Conversational AI Could Facilitate Thoughtful Reflection and Substantive Revision in Writingcited by 0
  12. 2025Dual Information Speech Language Models for Emotional Conversationscited by 0
  13. 2025Plug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognitioncited by 0
  14. 2025MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languagescited by 0
  15. 2026ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generationcited by 0
  16. 2026RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuancecited by 0
  17. 2026ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoningcited by 0
  18. 2026When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removalcited by 0
  19. 2026FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMscited by 0
  20. 2026CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agentscited by 0
  21. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0
  22. 2026Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Textcited by 0
  23. 2026ER-EDF: A Psychology-Grounded Emotion Regulation Framework for Speech Empathetic Dialogue Generation in Large Audio-Language Modelscited by 0
  24. 2026From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Modelscited by 0