mlsanityData to 7 Oct 2026

What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning

arXiv 2311.01487 ↗First posted 2 Nov 2023cs.CV cs.CLcited by 28

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Visual instruction tuning is crucial for enhancing the zero-shot generalization capability of Multi-modal Large Language Models (MLLMs). In this paper, we aim to investigate a fundamental question: ''what makes for good visual instructions''. Through a comprehensive empirical study, we find that instructions focusing on complex visual reasoning tasks are particularly effective in improving the performance of MLLMs, with results correlating to instruction complexity. Based on this insight, we develop a systematic approach to automatically create high-quality complex visual reasoning instructions. Our approach employs a synthesize-complicate-reformulate paradigm, leveraging multiple stages to gradually increase the complexity of the instructions while guaranteeing quality. Based on this approach, we create the ComVint dataset with 32K examples, and fine-tune four MLLMs on it. Experimental results consistently demonstrate the enhanced performance of all compared MLLMs, such as a 27.86% and 27.60% improvement for LLaVA on MME-Perception and MME-Cognition, respectively. Our code and data are publicly available at the link: https://github.com/RUCAIBox/ComVint.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 50 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

45 of 50 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
33 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  13. 2023A Survey on Multimodal Large Language Modelscited by 898
  14. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  15. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  16. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  18. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  20. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  21. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  22. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  23. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  24. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  25. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  26. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  27. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  28. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  29. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  30. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  31. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  32. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  33. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarkscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 12; 2026: 3.

28 papers.

Sort:

By area:

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  2. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  3. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
  4. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  5. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  6. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  7. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  8. 2024Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuningcited by 27
  9. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Modelcited by 21
  10. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  11. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  12. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
16 more
  1. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  2. 2025FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensicscited by 11
  3. 2026SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenescited by 11
  4. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 10
  5. 2026Towards Long-horizon Agentic Multimodal Searchcited by 9
  6. 2024EACO: Enhancing Alignment in Multimodal LLMs via Critical Observationcited by 8
  7. 2025NAVIG: Natural Language-guided Analysis with Vision Language Models for Image Geo-localizationcited by 8
  8. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Modelscited by 1
  9. 2025Evaluating Graphical Perception with Multimodal LLMscited by 1
  10. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  11. 2025CIMR: Contextualized Iterative Multimodal Reasoning for Robust Instruction Following in LVLMscited by 1
  12. 2026InEdit-Bench: Benchmarking Intermediate Logical Pathways for Intelligent Image Editing Modelscited by 1
  13. 2024C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learningcited by 0
  14. 2025Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructionscited by 0
  15. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Modelscited by 0
  16. 2025Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance
  2. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  3. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMs
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
  5. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  6. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
  7. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  8. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct