mlsanityData to 7 Oct 2026

VITA: Towards Open-Source Interactive Omni Multimodal LLM

arXiv 2408.05211 ↗First posted 9 Aug 2024cs.CV cs.AI cs.CLcited by 172

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research. Project Page: https://vita-home.github.io.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 62 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

53 of 62 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  4. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023A Survey on Multimodal Large Language Models23 Jun
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Mistral 7Bcited by 3,065
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
30 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  13. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  14. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  15. 2021InfographicVQAcited by 430
  16. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  17. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  18. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  19. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  20. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  21. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  22. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  23. 2023AnyText: Multilingual Visual Text Generation And Editingcited by 165
  24. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  25. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  26. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  27. 2024Language Model Can Listen While Speakingcited by 62
  28. 2019AI2D-RST: A multimodal corpus of 1000 primary school science diagramscited by 60
  29. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  30. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 98; 2026: 33.

172 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  3. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  4. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  5. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  6. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  7. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  8. 2025Thyme: Think Beyond Imagescited by 160
  9. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  10. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  12. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
38 more
  1. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  2. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  3. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  4. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  5. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  6. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  7. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  8. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  9. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 80
  10. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  11. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  12. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  13. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  14. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  15. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  16. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  17. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  18. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 57
  19. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  20. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  21. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  22. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  23. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
  24. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  25. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  26. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  27. 2024Baichuan-Omni Technical Reportcited by 42
  28. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  29. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  30. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  31. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  32. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  33. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  34. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  35. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  36. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  37. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  38. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generation
  2. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation
  3. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
  4. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  5. 2024PALO: A Polyglot Large Multimodal Model for 5B People
  6. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
  7. 2025Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction
  8. 2026Toward Native Multimodal Modeling: A Roadmap