mlsanityData to 7 Oct 2026

Baichuan-Omni Technical Report

arXiv 2410.08565 ↗First posted 11 Oct 2024cs.AI cs.CL cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

The salient multimodal capabilities and interactive experience of GPT-4o highlight its critical role in practical applications, yet it lacks a high-performing open-source counterpart. In this paper, we introduce Baichuan-omni, the first open-source 7B Multimodal Large Language Model (MLLM) adept at concurrently processing and analyzing modalities of image, video, audio, and text, while delivering an advanced multimodal interactive experience and strong performance. We propose an effective multimodal training schema starting with 7B model and proceeding through two stages of multimodal alignment and multitask fine-tuning across audio, image, video, and text modal. This approach equips the language model with the ability to handle visual and audio data effectively. Demonstrating strong performance across various omni-modal and multimodal benchmarks, we aim for this contribution to serve as a competitive baseline for the open-source community in advancing multimodal understanding and real-time interaction.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 83 of its 102 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

83 of 102 references.

References · 83

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  15. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  16. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  17. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  20. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  21. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  22. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  23. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  24. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  25. 2023A Survey on Multimodal Large Language Modelscited by 898
  26. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  27. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  28. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  29. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  30. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  31. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  32. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  33. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  34. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  35. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  36. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  37. 2024Qwen2-Audio Technical Reportcited by 543
  38. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 21; 2026: 18.

42 papers.

Sort:

By area:

  1. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  2. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  3. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  4. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
  5. 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
  6. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  7. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  8. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  9. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  10. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  11. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  12. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
30 more
  1. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  2. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  3. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  4. 2025Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentationcited by 10
  5. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  6. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  7. 2026Covo-Audio Technical Reportcited by 7
  8. 2024ChipAlign: Instruction Alignment in Large Language Models for Chip Design via Geodesic Interpolationcited by 6
  9. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  10. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  11. 2026AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinkingcited by 5
  12. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  13. 2025Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Modelscited by 4
  14. 2025A Structured Review of Underwater Object Detection Challenges and Solutions: From Traditional to Large Vision Language Modelscited by 4
  15. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  16. 2025Training-Free Multimodal Large Language Model Orchestrationcited by 2
  17. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understandingcited by 2
  18. 2026EgoAVU: Egocentric Audio-Visual Understandingcited by 2
  19. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 2
  20. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 2
  21. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  22. 2026Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoningcited by 1
  23. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Modelscited by 0
  24. 2025RealTalk-CN: A Realistic Chinese Speech-Text Dialogue Benchmark With Cross-Modal Interaction Analysiscited by 0
  25. 2026RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-trainingcited by 0
  26. 2026OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lensescited by 0
  27. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  28. 2026RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigationcited by 0
  29. 2026OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoningcited by 0
  30. 2026OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  2. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generation
  3. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  4. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  5. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2023Multimodal Large Language Models: A Survey
  8. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction