mlsanityData to 7 Oct 2026

Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models

arXiv 2408.01003 ↗First posted 2 Aug 2024cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made significant progress in bridging the gap between visual and language modalities. However, hallucinations in MLLMs, where the generated text does not align with image content, continue to be a major challenge. Existing methods for addressing hallucinations often rely on instruction-tuning, which requires retraining the model with specific data, which increases the cost of utilizing MLLMs further. In this paper, we introduce a novel training-free method, named Piculet, for enhancing the input representation of MLLMs. Piculet leverages multiple specialized models to extract descriptions of visual information from the input image and combine these descriptions with the original image and query as input to the MLLM. We evaluate our method both quantitively and qualitatively, and the results demonstrate that Piculet greatly decreases hallucinations of MLLMs. Our method can be easily extended to different MLLMs while being universal.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 34 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

29 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
  5. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
12 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2023A Survey on Multimodal Large Language Modelscited by 898
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2022PP-YOLOE: An evolved version of YOLOcited by 75
  12. 2016Deeply-Fused Netscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Hallucination of Multimodal Large Language Models: A Survey
  2. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  3. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
  4. 2024A Survey on Hallucination in Large Vision-Language Models
  5. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
  6. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  7. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens