mlsanityData to 14 Oct 2026

DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer

arXiv 2407.15130 ↗First posted 21 Jul 2024cs.CL cs.AIcited by 27

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

In this work, we introduce DOPRA, a novel approach designed to mitigate hallucinations in multi-modal large language models (MLLMs). Unlike existing solutions that typically involve costly supplementary training data or the integration of external knowledge sources, DOPRA innovatively addresses hallucinations by decoding specific weighted layer penalties and redistribution, offering an economical and effective solution without additional resources. DOPRA is grounded in unique insights into the intrinsic mechanisms controlling hallucinations within MLLMs, especially the models' tendency to over-rely on a subset of summary tokens in the self-attention matrix, neglecting critical image-related information. This phenomenon is particularly pronounced in certain strata. To counteract this over-reliance, DOPRA employs a strategy of weighted overlay penalties and redistribution in specific layers, such as the 12th layer, during the decoding process. Furthermore, DOPRA includes a retrospective allocation process that re-examines the sequence of generated tokens, allowing the algorithm to reallocate token selection to better align with the actual image content, thereby reducing the incidence of hallucinatory descriptions in auto-generated captions. Overall, DOPRA represents a significant step forward in improving the output quality of MLLMs by systematically reducing hallucinations through targeted adjustments during the decoding process.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 55 references that are in the map. First posted on arXiv 21 Jul 2024.

Builds on

32 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  6. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug

Other references · 26

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
14 more
  1. 2023How Language Model Hallucinations Can Snowballcited by 311
  2. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  3. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  4. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  5. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  6. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  7. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 77
  8. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  9. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
  10. 2023Large Language Models are Complex Table Parserscited by 28
  11. 2023A Large-scale Film Style Dataset for Learning Multi-frequency Driven Film Enhancementcited by 22
  12. 2023Brain Diffuser: An End-to-End Brain Image to Brain Network Pipelinecited by 21
  13. 2022ShaDocNet: Learning Spatial-Aware Tokens in Transformer for Document Shadow Removalcited by 16
  14. 2022WavEnhancer: Unifying Wavelet and Transformer for Image Enhancementcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  4. 2025Self-Evolving Multi-Agent Systems via Textual Backpropagationcited by 23
  5. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMscited by 15
  6. 2024Underwater Image Restoration via Polymorphic Large Kernel CNNscited by 14
  7. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 12
  8. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  9. 2024Test-Time Intensity Consistency Adaptation for Shadow Detectioncited by 11
  10. 2026Hallucination Begins Where Saliency Dropscited by 9
  11. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  12. 2025Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbationscited by 7
15 more
  1. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
  2. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Modelscited by 4
  3. 2025MAC-Lookup: Multi-Axis Conditional Lookup Model for Underwater Image Enhancementcited by 2
  4. 2025TextMamba: Scene Text Detector with Mambacited by 2
  5. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  6. 2025Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluationcited by 1
  7. 2026ART: Attention Replacement Technique to Improve Factuality in LLMscited by 1
  8. 2026Diffusion-CAM: Faithful Visual Explanations for dMLLMscited by 1
  9. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  10. 2025SFormer: SNR-guided Transformer for Underwater Image Enhancement from the Frequency Domaincited by 0
  11. 2025A Comprehensive Analysis of Churn Prediction in Telecommunications Using Machine Learningcited by 0
  12. 2025Forestpest-YOLO: A High-Performance Detection Framework for Small Forestry Pestscited by 0
  13. 2026DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Modelscited by 0
  14. 2026VEDAL: Variational Error-Driven Asynchronous Learning for 3D Gaussian Splatting Pruningcited by 0
  15. 2026MLLMs Hallucinate when Information Distribution Drifts in Synergy Headscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
  2. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding
  3. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  4. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  5. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs
  6. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
  7. 2024NoiseBoost: Alleviating Hallucination with Noise Perturbation for Multimodal Large Language Models
  8. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation