mlsanityData to 7 Oct 2026

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

arXiv 2412.05271 ↗First posted 6 Dec 2024cs.CVcited by 1,573

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality. In this work, we delve into the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluations on a wide range of benchmarks, including multi-discipline reasoning, document understanding, multi-image / video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a 3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing strong potential for test-time scaling. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems. HuggingFace demo see https://huggingface.co/spaces/OpenGVLab/InternVL

Key facts. Cited by 1,573 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 165 of its 317 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

165 of 317 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  2. 2024CMM-Math: A Chinese Multimodal Math Dataset To Evaluate and Enhance the Mathematics Reasoning of Large Multimodal Models4 Sep
  3. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine11 Jul
  4. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning18 Jun
  5. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text10 Jun
  6. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  7. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning10 Aug

Other references · 155

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2016Layer Normalizationcited by 5,579
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  14. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  15. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  16. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  17. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  18. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  19. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  22. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  23. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  24. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  25. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  26. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  27. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  28. 2024Yi: Open Foundation Models by 01.AIcited by 706
  29. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  30. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  31. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  32. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  33. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  34. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  35. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  36. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  37. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  38. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498

50 most cited of 155 All 155 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 1014; 2026: 545.

1,573 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  5. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  6. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  7. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  8. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  9. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  10. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  11. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  12. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
38 more
  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  4. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  5. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  6. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  7. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  8. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  9. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  10. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  11. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  12. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  13. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  14. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  15. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  16. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  17. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  18. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  19. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  20. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  21. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  22. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  23. 2025PaddleOCR 3.0 Technical Reportcited by 136
  24. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  25. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  26. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  27. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  28. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  29. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  30. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  31. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  32. 2025AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoningcited by 108
  33. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  34. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 105
  35. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  36. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  37. 2025Latent Visual Reasoningcited by 99
  38. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94

50 most cited of 1,573 All 1,573 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  4. 2025Benchmarking Large and Small MLLMs
  5. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  6. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  8. 2024The Revolution of Multimodal Large Language Models: A Survey