mlsanityData to 7 Oct 2026

MiniCPM-V: A GPT-4V Level MLLM on Your Phone

arXiv 2408.01800 ↗First posted 3 Aug 2024cs.CVcited by 1,061

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone. However, significant challenges remain preventing MLLMs from being practical in real-world applications. The most notable challenge comes from the huge cost of running an MLLM with a massive number of parameters and extensive computation. As a result, most MLLMs need to be deployed on high-performing cloud servers, which greatly limits their application scopes such as mobile, offline, energy-sensitive, and privacy-protective scenarios. In this work, we present MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By integrating the latest MLLM techniques in architecture, pretraining and alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1) Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong OCR capability and 1.8M pixel high-resolution image perception at any aspect ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual support for 30+ languages, and (5) efficient deployment on mobile phones. More importantly, MiniCPM-V can be viewed as a representative example of a promising trend: The model sizes for achieving usable (e.g., GPT-4V) level performance are rapidly decreasing, along with the fast growth of end-side computation capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices are becoming increasingly possible, unlocking a wider spectrum of real-world AI applications in the near future.

Key facts. Cited by 1,061 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 122 references that are in the map. First posted on arXiv 3 Aug 2024.

Builds on

52 of 122 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  5. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning2 Nov
  8. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants1 Oct
  9. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
28 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  9. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  10. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  11. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  12. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  13. 2024What matters when building vision-language models?cited by 321
  14. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  15. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  16. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  17. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  18. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  19. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  20. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  21. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  22. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  23. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  24. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  25. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  26. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  27. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  28. 2016The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)cited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 163; 2025: 598; 2026: 300.

1,061 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2025Qwen2.5-Omni Technical Reportcited by 704
  6. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  7. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  8. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  9. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  10. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  11. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  12. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
38 more
  1. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  2. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  3. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  4. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  5. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  7. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  8. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  9. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  10. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  11. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  12. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  13. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  14. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  15. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  16. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  17. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  18. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  19. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  20. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  21. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  22. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  23. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  24. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  25. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  26. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  27. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  28. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  29. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  30. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  31. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  32. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  33. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  34. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  35. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  36. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  37. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  38. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112

50 most cited of 1,061 All 1,061 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Large and Small MLLMs
  2. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  3. 2023A Survey on Multimodal Large Language Models
  4. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
  5. 2025BlueLM-2.5-3B Technical Report
  6. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
  7. 2025DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving
  8. 2025Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R