mlsanityData to 7 Oct 2026

Baichuan-Omni-1.5 Technical Report

arXiv 2501.15368 ↗First posted 26 Jan 2025cs.CL cs.SD eess.AScited by 107

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 151 of its 184 references that are in the map. First posted on arXiv 26 Jan 2025.

Builds on

151 of 184 references.

References · 151

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  13. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  14. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  18. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  19. 2023A Survey on Multimodal Large Language Modelscited by 898
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  22. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  23. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  24. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  25. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  26. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  27. 2022High Fidelity Neural Audio Compressioncited by 673
  28. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  29. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  30. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  31. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  32. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  33. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  34. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  35. 2024Qwen2-Audio Technical Reportcited by 543
  36. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  37. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  38. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436

50 most cited of 151 All 151 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 54; 2026: 53.

107 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2025Thyme: Think Beyond Imagescited by 160
  3. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  4. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  5. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  6. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  8. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  9. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  10. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  11. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  12. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
38 more
  1. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  2. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  3. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  4. 2025Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliancecited by 23
  5. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  6. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  7. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  8. 2025Fun-Audio-Chat Technical Reportcited by 17
  9. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  10. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  11. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  12. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  13. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Modelscited by 14
  14. 2025SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Modelscited by 14
  15. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  16. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  17. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  18. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 12
  19. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  20. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 10
  21. 2025TeleEgo: Benchmarking Egocentric AI Assistants in the Wildcited by 9
  22. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  23. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  24. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluencycited by 7
  25. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  26. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  27. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
  28. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  29. 2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
  30. 2025Pay More Attention To Audio: Mitigating Imbalance of Cross-Modal Attention in Large Audio Language Modelscited by 5
  31. 2025FysicsWorld: A Unified Full-Modality Benchmark for Any-to-Any Understanding, Generation, and Reasoningcited by 5
  32. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  33. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  34. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  35. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chainscited by 5
  36. 2025OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playingcited by 4
  37. 2025EchoX: Towards Mitigating Acoustic-Semantic Gap via Echo Training for Speech-to-Speech LLMscited by 4
  38. 2025XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Modelscited by 4

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
  2. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  3. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generation
  4. 2026Qwen3.5-Omni Technical Report
  5. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
  6. 2026Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
  7. 2025Qwen3-Omni Technical Report
  8. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model