mlsanityData to 7 Oct 2026

Visual Instruction Tuning with Polite Flamingo

arXiv 2307.01003 ↗First posted 3 Jul 2023cs.CV cs.CLcited by 38

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent research has demonstrated that the multi-task fine-tuning of multi-modal Large Language Models (LLMs) using an assortment of annotated downstream vision-language datasets significantly enhances their performance. Yet, during this process, a side effect, which we termed as the "multi-modal alignment tax", surfaces. This side effect negatively impacts the model's ability to format responses appropriately -- for instance, its "politeness" -- due to the overly succinct and unformatted nature of raw annotations, resulting in reduced human preference. In this paper, we introduce Polite Flamingo, a multi-modal response rewriter that transforms raw annotations into a more appealing, "polite" format. Polite Flamingo is trained to reconstruct high-quality responses from their automatically distorted counterparts and is subsequently applied to a vast array of vision-language datasets for response rewriting. After rigorous filtering, we generate the PF-1M dataset and further validate its value by fine-tuning a multi-modal LLM with it. Combined with novel methodologies including U-shaped multi-stage tuning and multi-turn augmentation, the resulting model, Clever Flamingo, demonstrates its advantages in both multi-modal understanding and response politeness according to automated and human evaluations.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 82 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

59 of 82 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A Survey on Multimodal Large Language Models23 Jun
  2. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023A Survey of Large Language Modelscited by 2,537
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  14. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  15. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  16. 2023Instruction Tuning with GPT-4cited by 676
  17. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  18. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  19. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  20. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  23. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  24. 2022Visual Spatial Reasoningcited by 288
  25. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  26. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  27. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  28. 2023Improving CLIP Training with Language Rewritescited by 216
  29. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  30. 2017Exploring Models and Data for Remote Sensing Image Caption Generationcited by 195
  31. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  32. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  33. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  34. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  35. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  36. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  37. 2022Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrievalcited by 77
  38. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 17; 2025: 9; 2026: 1.

38 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  5. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  6. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  7. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  8. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  9. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  10. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  11. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  12. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
26 more
  1. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 26
  2. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  3. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  4. 2024Enhancing Perception of Key Changes in Remote Sensing Image Change Captioningcited by 16
  5. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  6. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Modelscited by 12
  7. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  8. 2024What Makes for Good Image Captions?cited by 11
  9. 2024X-LLaVA: Optimizing Bilingual Large Vision-Language Alignmentcited by 10
  10. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
  11. 2025From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibrationcited by 8
  12. 2024Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!cited by 7
  13. 2024Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignmentcited by 6
  14. 2025A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimulicited by 6
  15. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  16. 2024Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imaginationcited by 5
  17. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5
  18. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  19. 2024MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosiscited by 4
  20. 2026Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integrationcited by 3
  21. 2024Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generationcited by 2
  22. 2025Playing Devil's Advocate: Unmasking Toxicity and Vulnerabilities in Large Vision-Language Modelscited by 2
  23. 2025Spoken Language Understanding on Unseen Tasks With In-Context Learningcited by 2
  24. 2025Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversioncited by 2
  25. 2024Exploring Compositional Generalization of Multimodal LLMs for Medical Imagingcited by 1
  26. 2023On the Evaluation and Refinement of Vision-Language Instruction Tuning Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models
  2. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics
  3. 2025Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning
  4. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  5. 2024Align2LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
  6. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
  7. 2025Improving Alignment in LVLMs with Debiased Self-Judgment
  8. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMs