mlsanityData to 7 Oct 2026

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

arXiv 2405.20797 ↗First posted 31 May 2024cs.CV cs.AI cs.CLcited by 167

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Current Multimodal Large Language Models (MLLMs) typically integrate a pre-trained LLM with another pre-trained vision transformer through a connector, such as an MLP, endowing the LLM with visual capabilities. However, the misalignment between two embedding strategies in MLLMs -- the structural textual embeddings based on an embedding look-up table and the continuous embeddings generated directly by the vision encoder -- makes challenges for a more seamless fusion of visual and textual information. We propose Ovis, a novel MLLM architecture designed to structurally align visual and textual embeddings. Ovis integrates an additional learnable visual embedding table into the visual encoder's process. To capture rich visual semantics, each image patch indexes the visual embedding table multiple times, resulting in a final visual embedding that is a probabilistic combination of the indexed embeddings. This structural approach mirrors the method used for generating textual embeddings. Empirical evaluations on various multimodal benchmarks show that Ovis outperforms open-source MLLMs of similar parameter scales and even surpasses the proprietary model Qwen-VL-Plus overall. These results highlight the potential of Ovis' structured visual representation for advancing MLLM architectural design and promoting more effective multimodal learning. Code, datasets, and models are available at https://github.com/AIDC-AI/Ovis.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 80 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

62 of 80 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  5. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  8. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023A Survey on Multimodal Large Language Models23 Jun
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  13. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  17. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  18. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  19. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  20. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  21. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  22. 2021OCR-free Document Understanding Transformercited by 400
  23. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  24. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  25. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  26. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  27. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  28. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  29. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  30. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  31. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  32. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  33. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  34. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  35. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  36. 2018Patch-Based Image Inpainting with Generative Adversarial Networkscited by 70
  37. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  38. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 116; 2026: 33.

167 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  5. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  6. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  7. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  8. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  9. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  10. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  11. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  12. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
38 more
  1. 2025Ovis-U1 Technical Reportcited by 65
  2. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  3. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 41
  4. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  5. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  6. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Trainingcited by 37
  7. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  8. 2025DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understandingcited by 35
  9. 2025AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Drivingcited by 35
  10. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  11. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  12. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  13. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 28
  14. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  15. 2025ShotBench: Expert-Level Cinematic Understanding in Vision-Language Modelscited by 22
  16. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMscited by 21
  17. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 20
  18. 2025Referring to Any Personcited by 20
  19. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  20. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  21. 2025AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoningcited by 19
  22. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  23. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17
  24. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  25. 2025Compositional Generative Model of Unbounded 4D Citiescited by 16
  26. 2025MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoningcited by 16
  27. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  28. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  29. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  30. 2025M2-Reasoning: Empowering MLLMs with Unified General and Spatial Reasoningcited by 16
  31. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  32. 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
  33. 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMscited by 14
  34. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Modelscited by 14
  35. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  36. 2025Multimodal Tabular Reasoning with Privileged Structured Informationcited by 14
  37. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  38. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curationcited by 13

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  2. 2025Visual Representation Alignment for Multimodal Large Language Models
  3. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
  4. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  5. 2024OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects
  6. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  7. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  8. 2025AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding