mlsanityData to 7 Oct 2026

EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

arXiv 2409.18042 ↗First posted 26 Sep 2024cs.CV cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging for the open-source community. Existing vision-language models rely on external tools for speech processing, while speech-language models still suffer from limited or totally without vision-understanding capabilities. To address this gap, we propose the EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech abilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we surprisingly notice that omni-modal alignment can further enhance vision-language and speech abilities compared with the bi-modal aligned counterparts. Moreover, a lightweight style module is introduced for the flexible speech style controls including emotions and pitches. For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 81 of its 105 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

81 of 105 references.

References · 81

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  13. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  14. 2016A Diagram Is Worth A Dozen Imagescited by 793
  15. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  16. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  17. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  18. 2024Qwen2-Audio Technical Reportcited by 543
  19. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  20. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  21. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  22. 2021InfographicVQAcited by 430
  23. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  24. 2021OCR-free Document Understanding Transformercited by 400
  25. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  26. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  27. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  28. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  29. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  30. 2024What matters when building vision-language models?cited by 321
  31. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  32. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  33. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  34. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  35. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  36. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  37. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  38. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 25; 2026: 9.

44 papers.

Sort:

By area:

  1. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  2. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  3. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  4. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  5. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  6. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  7. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  8. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  9. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  10. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  11. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  12. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
32 more
  1. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  2. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  3. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  4. 2024AtomThink: Multimodal Slow Thinking with Atomic Step Reasoningcited by 16
  5. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  6. 2025InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-trainingcited by 11
  7. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  8. 2025Can Atomic Step Decomposition Enhance the Self-structured Reasoning of Multimodal Large Models?cited by 10
  9. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  10. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  11. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  12. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 9
  13. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  14. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Modelscited by 7
  15. 2024MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Modelscited by 7
  16. 2025LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLMcited by 7
  17. 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interactioncited by 7
  18. 2025EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysiscited by 6
  19. 2025ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Drivingcited by 5
  20. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  21. 2025EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMscited by 4
  22. 2026Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimizationcited by 4
  23. 2025Investigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Modelscited by 3
  24. 2025End-to-end Listen, Look, Speak and Actcited by 3
  25. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Modelcited by 2
  26. 2026DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusioncited by 1
  27. 2026StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Controlcited by 1
  28. 2026Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learningcited by 1
  29. 2025Chain-of-Description: What I can understand, I can put into wordscited by 0
  30. 2026A2-LLM: An End-to-end Conversational Audio Avatar Large Language Modelcited by 0
  31. 2026Dynin-Robotics: Omnimodal Unified Diffusion Vision-Language-Action Modelcited by 0
  32. 2026EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
  2. 2025Learning to Hear by Seeing: It's Time for Vision Language Models to Understand Artistic Emotion from Sight and Sound
  3. 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMs
  4. 2025OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs
  5. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
  6. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  7. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
  8. 2025VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing