mlsanityData to 14 Oct 2026

InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

arXiv 2504.10479 ↗First posted 14 Apr 2025cs.CVcited by 1,834

Not on the map. Why?

Cited by 992 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce InternVL3, a significant advancement in the InternVL series featuring a native multimodal pre-training paradigm. Rather than adapting a text-only large language model (LLM) into a multimodal large language model (MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and linguistic capabilities from both diverse multimodal data and pure-text corpora during a single pre-training stage. This unified training paradigm effectively addresses the complexities and alignment challenges commonly encountered in conventional post-hoc training pipelines for MLLMs. To further improve performance and scalability, InternVL3 incorporates variable visual position encoding (V2PE) to support extended multimodal contexts, employs advanced post-training techniques such as supervised fine-tuning (SFT) and mixed preference optimization (MPO), and adopts test-time scaling strategies alongside an optimized training infrastructure. Extensive empirical evaluations demonstrate that InternVL3 delivers superior performance across a wide range of multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its capabilities remain highly competitive with leading proprietary models, including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also maintaining strong pure-language proficiency. In pursuit of open-science principles, we will publicly release both the training data and model weights to foster further research and development in next-generation MLLMs.

Key facts. Cited by 1,834 papers in the map (data through 14 Oct 2026). Builds on 128 of its 150 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

128 of 150 references.

References · 128

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  13. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  14. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  15. 2019Towards VQA Models That Can Readcited by 1,718
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  17. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  18. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  19. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  20. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  21. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  22. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  23. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  24. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  25. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  26. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  27. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  28. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  29. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  30. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  31. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  32. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  33. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  34. 2016A Diagram Is Worth A Dozen Imagescited by 804
  35. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  36. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  37. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  38. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 786; 2026: 1048.

1,834 papers.

Sort:

By area:

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  2. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  3. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  5. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  6. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  7. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
  8. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  9. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  10. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  11. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
  12. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 170
38 more
  1. 2025Thyme: Think Beyond Imagescited by 167
  2. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  3. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
  4. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  5. 2025GRIT: Teaching MLLMs to Think with Imagescited by 125
  6. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  7. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  8. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 118
  9. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  10. 2025Ovis2.5 Technical Reportcited by 108
  11. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  12. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 99
  13. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 96
  14. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 89
  15. 2025Visual Spatial Tuningcited by 89
  16. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  17. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  18. 2025Scaling RL to Long Videoscited by 84
  19. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
  20. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  21. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
  22. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  23. 2026DeepSeek-OCR 2: Visual Causal Flowcited by 74
  24. 2025VGR: Visual Grounded Reasoningcited by 73
  25. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  26. 2025Detect Anything via Next Point Predictioncited by 73
  27. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 71
  28. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 69
  29. 2025Ovis-U1 Technical Reportcited by 67
  30. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  31. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  32. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  33. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  34. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
  35. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  36. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 58
  37. 2025AdaThinkDrive: Adaptive Thinking via Reinforcement Learning for Autonomous Drivingcited by 58
  38. 2026PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsingcited by 57

50 most cited of 1,834 All 1,834 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models
  3. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training
  4. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  5. 2024E5-V: Universal Embeddings with Multimodal Large Language Models
  6. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Models
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models