mlsanityData to 14 Oct 2026

MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets

arXiv 2308.12067 ↗First posted 23 Aug 2023cs.LG cs.AI cs.CLcited by 61

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models are typically trained in two stages: first pre-training on image-text pairs, and then fine-tuning using supervised vision-language instruction data. Recent studies have shown that large language models can achieve satisfactory results even with a limited amount of high-quality instruction-following data. In this paper, we introduce MM-LIMA, which is fine-tuned on a small dataset comprising only 200 examples, amounting to approximately 6% of the instruction-following data used in the alignment dataset for MiniGPT-4. To achieve this, we first propose several metrics to access the quality of multimodal instruction data. Based on these metrics, we present an effective and trainable data selector to automatically identify and filter low-quality vision-language data. By employing this method, MM-LIMA outperforms the original MiniGPT-4 on various evaluations. Overall, our findings demonstrate that less but high-quality instruction tuning data is efficient in enabling multimodal large language models to generate better output. Our code is available at https://github.com/waltonfuture/InstructionGPT-4.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 38 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

28 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2019Towards VQA Models That Can Readcited by 1,718
11 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2023LIMA: Less Is More for Alignmentcited by 1,100
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  8. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  9. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  10. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  11. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 33; 2025: 15; 2026: 2.

61 papers.

Sort:

By area:

  1. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  2. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  3. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  4. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  5. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  6. 2024TinyAgent: Function Calling at the Edgecited by 58
  7. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 51
  8. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  9. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  10. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  11. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  12. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
38 more
  1. 2024CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Followingcited by 33
  2. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  3. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  4. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 27
  5. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignmentcited by 26
  6. 2024Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filterscited by 25
  7. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  8. 2024Efficient and Scalable Estimation of Tool Representations in Vector Spacecited by 22
  9. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  10. 2023CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuningcited by 19
  11. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  12. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  13. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spacescited by 17
  14. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?cited by 16
  15. 2023Instruction Tuning with Human Curriculumcited by 15
  16. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  17. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsalcited by 14
  18. 2024LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultationcited by 14
  19. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbationcited by 14
  20. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 14
  21. 2024Chasing Random: Instruction Selection Strategies Fail to Generalizecited by 13
  22. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Modelscited by 12
  23. 2025Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuningcited by 12
  24. 2024Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Modelcited by 10
  25. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
  26. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  27. 2025Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Rankingcited by 10
  28. 2024Do Large Language Models Mirror Cognitive Language Processing?cited by 9
  29. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
  30. 2024Curriculum Learning with Quality-Driven Data Selectioncited by 7
  31. 2023Data Optimization in Deep Learning: A Surveycited by 6
  32. 2024Can I understand what I create? Self-Knowledge Evaluation of Large Language Modelscited by 6
  33. 2025Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigmcited by 6
  34. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  35. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5
  36. 2024Generative AI for Immersive Communication: The Next Frontier in Internet-of-Senses Through 6Gcited by 4
  37. 2024Structural-Entropy-Based Sample Selection for Efficient and Effective Learningcited by 4
  38. 2024ResoFilter: Fine-grained Synthetic Data Filtering for Large Language Models through Data-Parameter Resonance Analysiscited by 4

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM
  2. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  3. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
  4. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
  5. 2026M-MiniGPT4: Multilingual VLLM Alignment via Translated Data
  6. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  7. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
  8. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training