mlsanityData to 14 Oct 2026

DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback

arXiv 2311.10081 ↗First posted 16 Nov 2023cs.CV cs.CL cs.LGcited by 103

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

We present DRESS, a large vision language model (LVLM) that innovatively exploits Natural Language feedback (NLF) from Large Language Models to enhance its alignment and interactions by addressing two key limitations in the state-of-the-art LVLMs. First, prior LVLMs generally rely only on the instruction finetuning stage to enhance alignment with human preferences. Without incorporating extra feedback, they are still prone to generate unhelpful, hallucinated, or harmful responses. Second, while the visual instruction tuning data is generally structured in a multi-turn dialogue format, the connections and dependencies among consecutive conversational turns are weak. This reduces the capacity for effective multi-turn interactions. To tackle these, we propose a novel categorization of the NLF into two key types: critique and refinement. The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences. The refinement NLF offers concrete suggestions for improvement and is adopted to improve the interaction ability of the LVLMs-- which focuses on LVLMs' ability to refine responses by incorporating feedback in multi-turn interactions. To address the non-differentiable nature of NLF, we generalize conditional reinforcement learning for training. Our experimental results demonstrate that DRESS can generate more helpful (9.76%), honest (11.52%), and harmless (21.03%) responses, and more effectively learn from feedback during multi-turn interactions compared to SOTA LVMLs.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 77 of its 125 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

77 of 125 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models8 Sep
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends17 Oct
  3. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models23 May
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  6. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  8. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  9. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 68

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  14. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  15. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  19. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  20. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  21. 2022Red Teaming Language Models with Language Modelscited by 1,026
  22. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  23. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  24. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  25. 2018Object Hallucination in Image Captioningcited by 669
  26. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  27. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  28. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  29. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  30. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  31. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  32. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  33. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  34. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  35. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  36. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  37. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  38. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 41; 2025: 40; 2026: 20.

103 papers.

Sort:

By area:

  1. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  4. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  7. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  8. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
  9. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  10. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  11. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  12. 2024Safety of Multimodal Large Language Models on Images and Textscited by 75
38 more
  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  2. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
  4. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  5. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
  6. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 60
  7. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
  8. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 48
  9. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  10. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
  11. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  12. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  13. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 33
  14. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 29
  15. 2024EventGPT: Event Stream Understanding with Multimodal Large Language Modelscited by 28
  16. 2024The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?cited by 26
  17. 2024Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Modelcited by 26
  18. 2024Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Modelscited by 25
  19. 2025Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injectioncited by 25
  20. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Studycited by 23
  21. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
  22. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 23
  23. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 21
  24. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21
  25. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  26. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  27. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  28. 2024Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verificationcited by 17
  29. 2025VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gapcited by 17
  30. 2025Understanding and Rectifying Safety Perception Distortion in VLMscited by 17
  31. 2025Automating Steering for Safe Multimodal Large Language Modelscited by 17
  32. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  33. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  34. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  35. 2023ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representationcited by 13
  36. 2025SEA: Low-Resource Safety Alignment for Multimodal Large Language Models via Synthetic Embeddingscited by 11
  37. 2024FACTOID: FACtual enTailment fOr hallucInation Detectioncited by 10
  38. 2025USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Modelscited by 10

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LVLMs are Bad at Overhearing Human Referential Communication
  2. 2026FPCO-Dialog: A Multi-Turn False-Premise Benchmark for Correction and Cooperation in Vision-Language Models
  3. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
  4. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  5. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning
  6. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models
  7. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  8. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervision