mlsanityData to 7 Oct 2026

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

arXiv 2308.04152 ↗First posted 8 Aug 2023cs.CVcited by 76

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG to concentrate solely on the primary visual contents sufficient for caption generation, often neglecting other visual details. This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can infer and complete the missing details essential for comprehending demonstrative instructions. Further, we propose a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instructions. As for evaluation, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation on the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C. Our benchmark, code, and pre-trained models are available at https://github.com/DCDmllm/Cheetah.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 79 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

63 of 79 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Mask R-CNNcited by 7,259
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  12. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  13. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  14. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  15. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  16. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  18. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  19. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  20. 2016Visual Storytellingcited by 176
  21. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  22. 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 137
  23. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 132
  24. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  25. 2021WebQA: Multihop and Multimodal QAcited by 109
  26. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  27. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  28. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  29. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  30. 2022Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learningcited by 59
  31. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  32. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  33. 2019Expressing Visual Relationships via Languagecited by 49
  34. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  35. 2016The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narrativescited by 41
  36. 2019Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigationcited by 39
  37. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  38. 2023Explore the Power of Synthetic Data on Few-shot Object Detectioncited by 38

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 40; 2025: 21; 2026: 8.

76 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  6. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  7. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  8. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  9. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  10. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  11. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 49
  12. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
38 more
  1. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  2. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  3. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Toolcited by 35
  4. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modelingcited by 34
  5. 2024FlexCAD: Unified and Versatile Controllable CAD Generation with Fine-tuned Large Language Modelscited by 33
  6. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  7. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  8. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  9. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  10. 2024HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editingcited by 29
  11. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  12. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layercited by 27
  13. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  14. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokenscited by 27
  15. 2024Quality Assessment in the Era of Large Models: A Surveycited by 26
  16. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  17. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  18. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  19. 2024HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Modelscited by 19
  20. 2024Towards Unified Multimodal Editing with Enhanced Knowledge Collaborationcited by 18
  21. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  22. 2024Improving Large Models with Small models: Lower Costs and Better Performancecited by 14
  23. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
  24. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  25. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMscited by 12
  26. 2024Semantic Alignment for Multimodal Large Language Modelscited by 12
  27. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  28. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  29. 2025WiseEdit: Benchmarking Cognition- and Creativity-Informed Image Editingcited by 10
  30. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
  31. 2023Good Questions Help Zero-Shot Image Reasoningcited by 8
  32. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  33. 2025Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learningcited by 8
  34. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8
  35. 2026Hallucination Begins Where Saliency Dropscited by 8
  36. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7
  37. 2024Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refiningcited by 7
  38. 2024RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Modelscited by 7

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs
  2. 2023VIGC: Visual Instruction Generation and Correction
  3. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  4. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models
  5. 2025VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
  6. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
  7. 2025Text-Only Data Synthesis for Vision Language Model Training
  8. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?