mlsanityData to 14 Oct 2026

Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation

arXiv 2508.16188 ↗First posted 22 Aug 2025cs.CL cs.CV cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

We present an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. We explore multiple visual encoders and multimodal fusion strategies during pre-training to identify the most effective integration approach. Subsequent fine-tuning on emotion recognition and expressive dialogue tasks yields substantial gains over speech-only baselines (e.g., +5 F1 in emotion recognition). AVLM highlights the value of expressive visual information in guiding speech generation and offers a foundation for end-to-end multimodal conversational systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 53 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

32 of 53 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 914
  8. 2022High Fidelity Neural Audio Compressioncited by 687
  9. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  10. 2024Qwen2-Audio Technical Reportcited by 560
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
20 more
  1. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  3. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  4. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  5. 2025Kimi-Audio Technical Reportcited by 222
  6. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  7. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  8. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  9. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  10. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  11. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  12. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  13. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  14. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  15. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  16. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  17. 2023LMCodec: A Low Bitrate Speech Codec With Causal Transformer Modelscited by 9
  18. 2024ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesiscited by 9
  19. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  20. 2023Learning Emotional Representations from Imbalanced Speech Data for Speech Emotion Recognition and Emotional Text-to-Speechcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation
  2. 2025Taming Transformer for Emotion-Controllable Talking Face Generation
  3. 2024FlowVQTalker: High-Quality Emotional Talking Face Generation through Normalizing Flow and Quantization
  4. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
  5. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound
  6. 2026Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
  7. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech
  8. 2026Acoustic and Semantic Modeling of Emotion in Spoken Language ↗