Feedback-Enhanced Hallucination-Resistant Vision-Language Model for Real-Time Scene Understanding
On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs
Abstract (arXiv)
Real-time scene comprehension is a key advance in artificial intelligence, enhancing robotics, surveillance, and assistive tools. However, hallucination remains a challenge. AI systems often misinterpret visual inputs, detecting nonexistent objects or describing events that never happened. These errors, far from minor, threaten reliability in critical areas like security and autonomous navigation where accuracy is essential. Our approach tackles this by embedding self-awareness into the AI. Instead of trusting initial outputs, our framework continuously assesses them in real time, adjusting confidence thresholds dynamically. When certainty falls below a solid benchmark, it suppresses unreliable claims. Combining YOLOv5's object detection strength with VILA1.5-3B's controlled language generation, we tie descriptions to confirmed visual data. Strengths include dynamic threshold tuning for better accuracy, evidence-based text to reduce hallucination, and real-time performance at 18 frames per second. This feedback-driven design cuts hallucination by 37 percent over traditional methods. Fast, flexible, and reliable, it excels in applications from robotic navigation to security monitoring, aligning AI perception with reality.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 26 references that are in the map. First posted on arXiv 7 Apr 2025.
Builds on
16 of 26 references.
References · 16
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2022Generative Adversarial Networkscited by 5,627
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
- 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
4 more
- 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
- 2018How To Backdoor Federated Learningcited by 765
- 2018Analyzing Federated Learning through an Adversarial Lenscited by 351
- 2018Mitigating Sybils in Federated Learning Poisoningcited by 161
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
- 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
- 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinations
- 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
- 2026Hallucination-aware intermediate representation edit in large vision-language models
- 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
- 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering
- 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models