mlsanityData to 7 Oct 2026

OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM

arXiv 2510.15870 ↗First posted 17 Oct 2025cs.CV cs.AI cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, outperforms Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens - a 6 times reduction compared to Qwen2.5-Omni's 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 97 of its 115 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

97 of 115 references.

References · 97

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  13. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  14. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  15. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  16. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  17. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  18. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  19. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  20. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  21. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  22. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  23. 2016A Diagram Is Worth A Dozen Imagescited by 793
  24. 2025Qwen2.5-Omni Technical Reportcited by 704
  25. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  26. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  27. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  28. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  29. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  30. 2024Qwen2-Audio Technical Reportcited by 543
  31. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  32. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  33. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  34. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  35. 2018The UCR Time Series Archivecited by 421
  36. 2019Clotho: An Audio Captioning Datasetcited by 354
  37. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  38. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 50.

53 papers.

Sort:

By area:

  1. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 25
  2. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  3. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  4. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videoscited by 14
  5. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 12
  6. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  7. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8
  8. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 8
  9. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  10. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  11. 2026OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimizationcited by 5
  12. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chainscited by 5
38 more
  1. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 3
  2. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 3
  3. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  4. 2026MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generationcited by 3
  5. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  6. 2026The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMscited by 2
  7. 2026Dynin-Omni: Omnimodal Unified Large Diffusion Language Modelcited by 2
  8. 2026Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Modelscited by 2
  9. 2026OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Modelscited by 2
  10. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  11. 2026AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compressioncited by 2
  12. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  13. 2026FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliancecited by 1
  14. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  15. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
  16. 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Modelscited by 1
  17. 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
  18. 2026CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scalescited by 1
  19. 2026SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Modelscited by 1
  20. 2026OmniReasoner: Thinking with Long Audio-Video via Native Tool Usecited by 1
  21. 2026OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Modelscited by 1
  22. 2026Allocation Before Ranking: Decoupled Token Compression for OmniLLMscited by 1
  23. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  24. 2026Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoningcited by 1
  25. 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflictcited by 1
  26. 2026OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agentscited by 1
  27. 2026Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMscited by 1
  28. 2025OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusioncited by 0
  29. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  30. 2026Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMscited by 0
  31. 2026OmniTrace: A Unified Framework for Generation-Time Attribution in Omni-Modal LLMscited by 0
  32. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0
  33. 2026Reinforcing Dual-Path Reasoning in Spatial Vision Language Modelscited by 0
  34. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0
  35. 2026Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activationcited by 0
  36. 2026Training-Free Speech-Centric Omni Understanding with Frozen VLMscited by 0
  37. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  38. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguecited by 0

50 most cited of 53 All 53 (JSON)