mlsanityData to 7 Oct 2026

VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

arXiv 2501.01957 ↗First posted 3 Jan 2025cs.CV cs.SD eess.AScited by 191

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction. However, speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge due to the fundamental modality differences. In this paper, we propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information, ultimately enabling fluent vision and speech interaction. Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules, significantly accelerating multimodal end-to-end response speed. By comparing our method against state-of-the-art counterparts across benchmarks for image, video, and speech tasks, we demonstrate that our model is equipped with both strong visual and speech capabilities, making near real-time vision and speech interaction. Code has been released at https://github.com/VITA-MLLM/VITA.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 78 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

63 of 78 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs22 Nov
  2. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM1 Nov
  3. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?23 Aug
  4. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM9 Aug
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  9. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  12. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model8 Oct
5 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2023Mistral 7Bcited by 3,065
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
34 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  6. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  7. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  10. 2024Long Context Transfer from Language to Visioncited by 502
  11. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  12. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  13. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  14. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  15. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  16. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  17. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  18. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  19. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  20. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  21. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  22. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  23. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  24. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  25. 2023Audio-Visual LLM for Video Understandingcited by 76
  26. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  27. 2019AI2D-RST: A multimodal corpus of 1000 primary school science diagramscited by 60
  28. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  29. 2023Fewer-token Neural Speech Codec with Time-invariant Codescited by 21
  30. 2020Geometry-guided Dense Perspective Network for Speech-Driven Facial Animationcited by 19
  31. 2023E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Modelscited by 19
  32. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
  33. 2022Improving Multimodal Speech Recognition by Data Augmentation and Speech Representationscited by 13
  34. 2023SpeechAct: Towards Generating Whole-body Motion from Speechcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 108; 2026: 83.

191 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025Thyme: Think Beyond Imagescited by 160
  3. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  4. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  6. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  7. 2025Scaling RL to Long Videoscited by 83
  8. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  9. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  10. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  11. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  12. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
38 more
  1. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 54
  2. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  3. 2025Kwai Keye-VL Technical Reportcited by 47
  4. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  5. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 46
  6. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  7. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  8. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  9. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 39
  10. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  11. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  12. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  13. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  14. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  15. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  16. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  17. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 28
  18. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 27
  19. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  20. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  21. 2025Efficient Vision-Language-Action Models for Embodied Manipulation: A Systematic Surveycited by 25
  22. 2025Streaming Video Instruction Tuningcited by 25
  23. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  24. 2025LongCat-Flash-Omni Technical Reportcited by 23
  25. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  26. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  27. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  28. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  29. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  30. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  31. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  32. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  33. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  34. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  35. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  36. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  37. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  38. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  2. 2025Vision-Speech Models: Teaching Speech Models to Converse about Images
  3. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  4. 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning
  5. 2025SVLA: A Unified Speech-Vision-Language Assistant with Multimodal Reasoning and Speech Generation
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  8. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI