mlsanityData to 7 Oct 2026

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

arXiv 2403.18814 ↗First posted 27 Mar 2024cs.CV cs.AI cs.CLcited by 361

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In this work, we introduce Mini-Gemini, a simple and effective framework enhancing multi-modality Vision Language Models (VLMs). Despite the advancements in VLMs facilitating basic visual dialog and reasoning, a performance gap persists compared to advanced models like GPT-4 and Gemini. We try to narrow the gap by mining the potential of VLMs for better performance and any-to-any workflow from three aspects, i.e., high-resolution visual tokens, high-quality data, and VLM-guided generation. To enhance visual tokens, we propose to utilize an additional visual encoder for high-resolution refinement without increasing the visual token count. We further construct a high-quality dataset that promotes precise image comprehension and reasoning-based generation, expanding the operational scope of current VLMs. In general, Mini-Gemini further mines the potential of VLMs and empowers current frameworks with image understanding, reasoning, and generation simultaneously. Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs) from 2B to 34B. It is demonstrated to achieve leading performance in several zero-shot benchmarks and even surpasses the developed private models. Code and models are available at https://github.com/dvlab-research/MiniGemini.

Key facts. Cited by 361 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 92 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

61 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  13. 2024Mixtral of Expertscited by 1,785
  14. 2019Towards VQA Models That Can Readcited by 1,696
  15. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  16. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  17. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  18. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  19. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  20. 2023LIMA: Less Is More for Alignmentcited by 1,080
  21. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  22. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  23. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  24. 2016A Diagram Is Worth A Dozen Imagescited by 793
  25. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  26. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  27. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  28. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  29. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  30. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  31. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  32. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  33. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  34. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  35. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  36. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  37. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  38. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 172; 2025: 147; 2026: 42.

361 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  8. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  9. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  10. 2024What matters when building vision-language models?cited by 321
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  12. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
38 more
  1. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  2. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  3. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  4. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  5. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  6. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  7. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  10. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  11. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  12. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  13. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  14. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  15. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  16. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  17. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  18. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  19. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  20. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  21. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  22. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  23. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  24. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  25. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  26. 2024VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understandingcited by 116
  27. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  28. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  29. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  30. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  31. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  32. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  33. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  34. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  35. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  36. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  37. 2024Benchmarking and Improving Detail Image Captioncited by 81
  38. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81

50 most cited of 361 All 361 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
  2. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  4. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  5. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding
  6. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  7. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding