mlsanityData to 7 Oct 2026

CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

arXiv 2309.02301 ↗First posted 5 Sep 2023cs.CVcited by 69

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Nowadays, the research on Large Vision-Language Models (LVLMs) has been significantly promoted thanks to the success of Large Language Models (LLM). Nevertheless, these Vision-Language Models (VLMs) are suffering from the drawback of hallucination -- due to insufficient understanding of vision and language modalities, VLMs may generate incorrect perception information when doing downstream applications, for example, captioning a non-existent entity. To address the hallucination phenomenon, on the one hand, we introduce a Contrastive Instruction Evaluation Method (CIEM), which is an automatic pipeline that leverages an annotated image-text dataset coupled with an LLM to generate factual/contrastive question-answer pairs for the evaluation of the hallucination of VLMs. On the other hand, based on CIEM, we further propose a new instruction tuning method called CIT (the abbreviation of Contrastive Instruction Tuning) to alleviate the hallucination of VLMs by automatically producing high-quality factual/contrastive question-answer pairs and corresponding justifications for model tuning. Through extensive experiments on CIEM and CIT, we pinpoint the hallucination issues commonly present in existing VLMs, the disability of the current instruction-tuning dataset to handle the hallucination phenomenon and the superiority of CIT-tuned VLMs over both CIEM and public datasets.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 28 references that are in the map. First posted on arXiv 5 Sep 2023.

Builds on

28 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
15 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2018Object Hallucination in Image Captioningcited by 661
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  12. 2018nocaps: novel object captioning at scalecited by 444
  13. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  14. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  15. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 29; 2025: 30; 2026: 9.

69 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  4. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  5. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  6. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  7. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  8. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  9. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  10. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  11. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27
  12. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 27
38 more
  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  2. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Modelscited by 19
  3. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decodingcited by 19
  4. 2024BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Modelscited by 18
  5. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  6. 2024Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verificationcited by 17
  7. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  8. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  9. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Modelscited by 15
  10. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Modelscited by 12
  11. 2024DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Modelscited by 11
  12. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  13. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questionscited by 10
  14. 2025Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Modelscited by 9
  15. 2025Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?cited by 9
  16. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?cited by 8
  17. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8
  18. 2024LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Modelscited by 7
  19. 2025VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?cited by 7
  20. 2025ChartCap: Mitigating Hallucination of Dense Chart Captioningcited by 7
  21. 2024Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuningcited by 6
  22. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMscited by 6
  23. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decodingcited by 6
  24. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answeringcited by 5
  25. 2024From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuningcited by 5
  26. 2024Trust but Verify: Programmatic VLM Evaluation in the Wildcited by 5
  27. 2025Understanding and Evaluating Hallucinations in 3D Visual Language Modelscited by 5
  28. 2025Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-trainingcited by 5
  29. 2024Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Modelscited by 4
  30. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Modelscited by 4
  31. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  32. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  33. 2025GHOST: Hallucination-Inducing Image Generation for Multimodal LLMscited by 4
  34. 2024FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphscited by 3
  35. 2024Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMscited by 3
  36. 2024Understanding Multimodal Hallucination with Parameter-Free Representation Alignmentcited by 2
  37. 2025Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScorecited by 2
  38. 2025TPC: Cross-Temporal Prediction Connection for Vision-Language Model Hallucination Reductioncited by 2

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  3. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
  4. 2025Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models
  5. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)
  6. 2024Automatically Generating Visual Hallucination Test Cases for Multimodal Large Language Models
  7. 2024C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learning
  8. 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models