mlsanityData to 7 Oct 2026

Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions

arXiv 2408.01091 ↗First posted 2 Aug 2024cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) excel in adhering to human instructions. However, self-contradictory instructions may arise due to the increasing trend of multimodal interaction and context length, which is challenging for language beginners and vulnerable populations. We introduce the Self-Contradictory Instructions benchmark to evaluate the capability of LMMs in recognizing conflicting commands. It comprises 20,000 conflicts, evenly distributed between language and vision paradigms. It is constructed by a novel automatic dataset creation framework, which expedites the process and enables us to encompass a wide range of instruction forms. Our comprehensive evaluation reveals current LMMs consistently struggle to identify multimodal instruction discordance due to a lack of self-awareness. Hence, we propose the Cognitive Awakening Prompting to inject cognition from external, largely enhancing dissonance detection. The dataset and code are here: https://selfcontradiction.github.io/.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 52 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

38 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts20 Feb
  2. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information31 Jan
  3. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  6. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
20 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  9. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  10. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  11. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  12. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  13. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  14. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  15. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  16. 2023Resolving Knowledge Conflicts in Large Language Modelscited by 49
  17. 2023Managing extreme AI risks amid rapid progresscited by 48
  18. 2023Propagating Knowledge Updates to LMs Through Distillationcited by 28
  19. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  20. 2023AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queriescited by 10
  2. 2025REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?cited by 7
  3. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  4. 2025Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separatorscited by 1
  5. 2026Causal Evidence for Attention Head Imbalance in Modality Conflict Hallucinationcited by 1
  6. 2026TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patternscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
  2. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
  3. 2026Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
  4. 2024MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception
  5. 2025When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning
  6. 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
  7. 2026The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning
  8. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models