mlsanityData to 7 Oct 2026

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

arXiv 2312.14238 ↗First posted 21 Dec 2023cs.CVcited by 2,534

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The exponential growth of large language models (LLMs) has opened up numerous possibilities for multimodal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs. In this work, we design a large-scale vision-language foundation model (InternVL), which scales up the vision foundation model to 6 billion parameters and progressively aligns it with the LLM, using web-scale image-text data from various sources. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image/video-text retrieval, and link with LLMs to create multi-modal dialogue systems. It has powerful visual capabilities and can be a good alternative to the ViT-22B. We hope that our research could contribute to the development of multi-modal large models. Code and models are available at https://github.com/OpenGVLab/InternVL.

Key facts. Cited by 2,534 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 146 of its 193 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

146 of 193 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  4. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  5. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models18 Sep
  6. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 136

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2022A ConvNet for the 2020scited by 3,545
  13. 2023Qwen Technical Reportcited by 3,273
  14. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  18. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  19. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  20. 2013Describing Textures in the Wildcited by 2,110
  21. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  22. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  23. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  24. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  25. 2019Towards VQA Models That Can Readcited by 1,696
  26. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  27. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  28. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  29. 2017Deformable Convolutional Networkscited by 1,481
  30. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  31. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  32. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  33. 2019Natural Adversarial Examplescited by 1,401
  34. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  35. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  36. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  37. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  38. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173

50 most cited of 136 All 136 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 524; 2025: 1221; 2026: 788.

2,534 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  11. 2024Long Context Transfer from Language to Visioncited by 502
  12. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
38 more
  1. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  2. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  4. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  5. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  6. 2024OmniGen: Unified Image Generationcited by 388
  7. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  8. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  9. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  11. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  12. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  13. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  14. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  15. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  16. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  17. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  18. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  19. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  20. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  21. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  22. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  23. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  24. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  25. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  26. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  27. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  28. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  29. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  30. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  31. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  32. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  33. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  34. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  35. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  36. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  37. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  38. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195

50 most cited of 2,534 All 2,534 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  5. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  6. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  8. 2024The Revolution of Multimodal Large Language Models: A Survey