mlsanityData to 14 Oct 2026

LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

arXiv 2311.11860 ↗First posted 20 Nov 2023cs.CVcited by 81

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have endowed LLMs with the ability to perceive and understand multi-modal signals. However, most of the existing MLLMs mainly adopt vision encoders pretrained on coarsely aligned image-text pairs, leading to insufficient extraction and reasoning of visual knowledge. To address this issue, we devise a dual-Level vIsual knOwledge eNhanced Multimodal Large Language Model (LION), which empowers the MLLM by injecting visual knowledge in two levels. 1) Progressive incorporation of fine-grained spatial-aware visual knowledge. We design a vision aggregator cooperated with region-level vision-language (VL) tasks to incorporate fine-grained spatial-aware visual knowledge into the MLLM. To alleviate the conflict between image-level and region-level VL tasks during incorporation, we devise a dedicated stage-wise instruction-tuning strategy with mixture-of-adapters. This progressive incorporation scheme contributes to the mutual promotion between these two kinds of VL tasks. 2) Soft prompting of high-level semantic visual evidence. We facilitate the MLLM with high-level semantic visual evidence by leveraging diverse image tags. To mitigate the potential influence caused by imperfect predicted tags, we propose a soft prompting method by embedding a learnable token into the tailored text instruction. Comprehensive experiments on several multi-modal benchmarks demonstrate the superiority of our model (e.g., improvement of 5% accuracy on VSR and 3% CIDEr on TextCaps over InstructBLIP, 5% accuracy on RefCOCOg over Kosmos-2).

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 58 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

52 of 58 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs1 Oct
  3. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  4. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  5. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  10. 2023LMEye: An Interactive Perception Network for Large Language Models5 May

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2021Grounded Language-Image Pre-trainingcited by 1,156
30 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  13. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  14. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  15. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  17. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  18. 2016Visual Dialogcited by 377
  19. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  20. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  21. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  22. 2022Visual Spatial Reasoningcited by 293
  23. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  24. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  25. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  26. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 99
  27. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 56
  28. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  29. 2022Open-set Adversarial Defense with Clean-Adversarial Mutual Learningcited by 21
  30. 2023GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regionscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 42; 2026: 19.

81 papers.

Sort:

By area:

  1. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  2. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  3. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  4. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  5. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  6. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  7. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  8. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  9. 2024Toward Visual Grounding: A Surveycited by 52
  10. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 46
  11. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  12. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
38 more
  1. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  2. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 28
  3. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  4. 2025Visual-Instructed Degradation Diffusion for All-in-One Image Restorationcited by 26
  5. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuningcited by 22
  6. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 22
  7. 2025Mind with Eyes: from Language Reasoning to Multimodal Reasoningcited by 22
  8. 2024The Evolution of Multimodal Model Architecturescited by 21
  9. 2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 21
  10. 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
  11. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
  12. 2025UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queriescited by 18
  13. 2025Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraintscited by 17
  14. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  15. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Informationcited by 16
  16. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  17. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  18. 2025STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantizationcited by 13
  19. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  20. 2025Visual Intention Grounding for Egocentric Assistantscited by 11
  21. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  22. 2024To Preserve or To Compress: An In-Depth Study of Connector Selection in Multimodal Large Language Modelscited by 10
  23. 2025PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discriminationcited by 10
  24. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  25. 2025Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptationcited by 9
  26. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 8
  27. 2025Improving Generalized Visual Grounding with Instance-aware Joint Learningcited by 8
  28. 2024LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Descriptioncited by 7
  29. 2025E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detectioncited by 7
  30. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  31. 2025Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Predictioncited by 7
  32. 2025Latent Expression Generation for Referring Image Segmentation and Groundingcited by 6
  33. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 6
  34. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Modelscited by 6
  35. 2025SmokeBench: Evaluating Multimodal Large Language Models for Wildfire Smoke Detectioncited by 5
  36. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  37. 2024OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objectscited by 3
  38. 2025RONA: Pragmatically Diverse Image Captioning with Coherence Relationscited by 3

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  2. 2023Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs
  3. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
  4. 2025Introducing Visual Perception Token into Multimodal Large Language Model
  5. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  6. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  7. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  8. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models