mlsanityData to 13 Oct 2026

How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

arXiv 2404.16821 ↗First posted 25 Apr 2024cs.CVcited by 1,041

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448×448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.

Key facts. Cited by 1,041 papers in the map (data through 13 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 115 of its 148 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

115 of 148 references.

Closest ancestors · 25

Same direction, newest first

  1. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  2. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models18 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models29 Mar
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  6. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  7. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  8. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
13 more
  1. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  2. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  3. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model8 Oct
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  6. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  7. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models21 Aug
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 90

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  8. 2023Visual Instruction Tuningcited by 7,827
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  12. 2023Qwen Technical Reportcited by 3,310
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  8. 2019Towards VQA Models That Can Readcited by 1,705
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,595
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,461
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,116
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,082
  15. 2016Modeling Context in Referring Expressionscited by 1,057
  16. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  17. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 898
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  19. 2016A Diagram Is Worth A Dozen Imagescited by 796
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 778
  21. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 773
  22. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 730
  23. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 726
  24. 2024Yi: Open Foundation Models by 01.AIcited by 715
  25. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 706
  26. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 706
  27. 2023CogAgent: A Visual Language Model for GUI Agentscited by 696
  28. 2023Baichuan 2: Open Large-scale Language Modelscited by 668
  29. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 634
  30. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  31. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 577
  32. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 517
  33. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  34. 2021InfographicVQAcited by 431
  35. 2021OCR-free Document Understanding Transformercited by 402
  36. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  37. 2024InternLM2 Technical Reportcited by 393
  38. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 378

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 328; 2025: 517; 2026: 196.

1,041 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,798
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,586
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,452
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,389
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,070
  7. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 872
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 768
  10. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 588
  11. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 520
  12. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 510
38 more
  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 508
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 502
  3. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 451
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 446
  5. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 406
  6. 2024OmniGen: Unified Image Generationcited by 392
  7. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 373
  8. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 323
  9. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 306
  10. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 299
  11. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 288
  12. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 283
  13. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 245
  14. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  15. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 233
  16. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 231
  17. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 214
  18. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 211
  19. 2024NVILA: Efficient Frontier Visual Language Modelscited by 208
  20. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  21. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 206
  22. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  23. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  24. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 192
  25. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  26. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  27. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  28. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 179
  29. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 178
  30. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 178
  31. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 177
  32. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 177
  33. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 173
  34. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  35. 2024Building and better understanding vision-language models: insights and future directionscited by 172
  36. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 171
  37. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  38. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 167

50 most cited of 1,041 All 1,041 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial
  2. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
  3. 2023Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning
  4. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
  5. 2025Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning
  6. 2024Cross-Modal Consistency in Multimodal Large Language Models
  7. 2024M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining
  8. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages