mlsanityData to 7 Oct 2026

Efficient Multimodal Learning from Data-centric Perspective

arXiv 2402.11530 ↗First posted 18 Feb 2024cs.CVcited by 120

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks. However, their deployment is hindered by substantial computational costs in both training and inference, limiting accessibility to the broader research and user communities. A straightforward solution is to leverage smaller pre-trained vision and language models, which inevitably cause significant performance drops. In this paper, we demonstrate the possibility of training a smaller but better MLLM with high-quality training data. Specifically, we introduce Bunny, a family of lightweight MLLMs with flexible vision and language backbones for efficient multimodal learning from selected training data. Experiments show that our Bunny-4B/8B outperforms the state-of-the-art large MLLMs on multiple benchmarks. We expect that this work can provide the community with a clean and flexible open-source tool for further research and development. The code, models, and data can be found in https://github.com/BAAI-DCAI/Bunny.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 63 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

42 of 63 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  2. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  3. 2023VILA: On Pre-training for Visual Language Models12 Dec
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2023Qwen Technical Reportcited by 3,273
19 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  14. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  15. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  16. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  17. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  18. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  19. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 64; 2025: 39; 2026: 17.

120 papers.

Sort:

By area:

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  5. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  6. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  7. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  8. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  9. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  10. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  11. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  12. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 95
38 more
  1. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  2. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  3. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  4. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  5. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  6. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  7. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  8. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  9. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  10. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  11. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  12. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
  13. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  14. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  15. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 42
  16. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  17. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  18. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 32
  19. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  20. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  21. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  22. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  23. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  24. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  25. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  26. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Modelscited by 24
  27. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  28. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  29. 2025LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimizationcited by 20
  30. 2024Imp: Highly Capable Large Multimodal Models for Mobile Devicescited by 19
  31. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  32. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Bettercited by 17
  33. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  34. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  35. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizercited by 15
  36. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 14
  37. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  38. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  3. 2025Benchmarking Large and Small MLLMs
  4. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  5. 2026MoR-MLLM: Mixture of Recursions for Efficient Multimodal Large Language Models
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  8. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models