mlsanityData to 14 Oct 2026

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

arXiv 2410.08202 ↗First posted 10 Oct 2024cs.CV cs.CLcited by 82

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we focus on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM. In particular, we identify that existing pre-training strategies for monolithic MLLMs often suffer from unstable optimization or catastrophic forgetting. To address this issue, our core idea is to embed a new visual parameter space into a pre-trained LLM, thereby stably learning visual knowledge from noisy data while freezing the LLM. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-training strategy to maximize the visual capability of Mono-InternVL, namely Endogenous Visual Pre-training (EViP). In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data. To validate our approach, we conduct extensive experiments on 16 benchmarks. Experimental results confirm the superior performance of Mono-InternVL than existing monolithic MLLMs on 13 of 16 multimodal benchmarks, e.g., +80 points over Emu3 on OCRBench. Compared to the modular baseline, i.e., InternVL-1.5, Mono-InternVL still retains comparable multimodal performance while reducing up to 67% first token latency. Code and model are released at https://github.com/OpenGVLab/Mono-InternVL.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 106 of its 132 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

106 of 132 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  5. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  10. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  11. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
5 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 89

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  13. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  16. 2016Modeling Context in Referring Expressionscited by 1,063
  17. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  20. 2016A Diagram Is Worth A Dozen Imagescited by 804
  21. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  22. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  23. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  24. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  25. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  26. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  27. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  28. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  29. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  30. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  31. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  32. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  33. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  34. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  35. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  36. 2021InfographicVQAcited by 437
  37. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  38. 2021OCR-free Document Understanding Transformercited by 407

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 49; 2026: 23.

82 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  5. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  6. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  7. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 99
  8. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learningcited by 71
  9. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 51
  10. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  11. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  12. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
38 more
  1. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  2. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  3. 2025Vision as LoRAcited by 37
  4. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  5. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  6. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  7. 2024A Survey of Small Language Modelscited by 31
  8. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 30
  9. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  10. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 28
  11. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 28
  12. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  13. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  14. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 23
  15. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  16. 2025GenExam: A Multidisciplinary Text-to-Image Examcited by 22
  17. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 20
  18. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Modelscited by 18
  19. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 18
  20. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  21. 2024TAR3D: Creating High-Quality 3D Assets via Next-Part Predictioncited by 16
  22. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
  23. 2024γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Modelscited by 14
  24. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  25. 2025AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessmentcited by 11
  26. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  27. 2025TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Modelscited by 9
  28. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  29. 2025Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understandingcited by 8
  30. 2025InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognitioncited by 8
  31. 2026From Pixels to Words -- Towards Native One-Vision Models at Scalecited by 8
  32. 2025BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queriescited by 7
  33. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forwardcited by 7
  34. 2026GRADE: Benchmarking Discipline-Informed Reasoning in Image Editingcited by 7
  35. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 7
  36. 2024Multimodal Music Generation with Explicit Bridges and Retrieval Augmentationcited by 6
  37. 2025ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolutioncited by 6
  38. 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMscited by 5

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  3. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  4. 2025Visual Representation Alignment for Multimodal Large Language Models
  5. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  6. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
  7. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  8. 2026Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution