mlsanityData to 7 Oct 2026

Towards Robust Instruction Tuning on Multimodal Large Language Models

arXiv 2402.14492 ↗First posted 22 Feb 2024cs.CL cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks. Recent works about high-quality instruction-following data generation and selection require amounts of human labor to conceive model-understandable instructions for the given tasks and carefully filter the LLM-generated data. In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. It starts from a handful of basic and straightforward meta instructions but can expand an instruction-following dataset by 30 times. Results on two popular multimodal instructionfollowing benchmarks MULTIINSTRUCT and InstructBLIP show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks, which is even equivalent to the benefits of scaling up training data multiple times.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 64 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

38 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
23 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  12. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  13. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  14. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  15. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  16. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  17. 2023Self-Alignment with Instruction Backtranslationcited by 149
  18. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  19. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  20. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  21. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  22. 2022MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequencescited by 8
  23. 2023Self-Adaptive Sampling for Efficient Video Question-Answering on Image--Text Modelscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Visual Instruction Bottleneck Tuningcited by 5
  2. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  2. 2025MM-IFEngine: Towards Multimodal Instruction Following
  3. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  4. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
  5. 2025Aligning Instruction Tuning with Pre-training
  6. 2025IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Models
  7. 2025HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model
  8. 2024CoMMIT: Coordinated Multimodal Instruction Tuning