HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restoration
Not on the map. Why?
Abstract (arXiv)
Generative models are prone to hallucinations: plausible but incorrect structures absent in the ground truth. This issue is problematic in image restoration for safety-critical domains such as medical imaging, industrial inspection, and remote sensing, where such errors undermine reliability and trust. For example, in low-field MRI, widely used in resource-limited settings, restoration models are essential for enhancing low-quality scans, yet hallucinations can lead to serious diagnostic errors. Progress has been hindered by a circular dependency: evaluating hallucinations requires labeled data, yet such labels are costly and subjective. We introduce HalluGen, a diffusion-based framework that synthesizes realistic hallucinations with controllable type, location, and severity, producing perceptually realistic but semantically incorrect outputs (segmentation IoU drops from 0.86 to 0.36). Using HalluGen, we construct the first large-scale hallucination dataset comprising 4,350 annotated images derived from 1,450 brain MR images for low-field enhancement, enabling systematic evaluation of hallucination detection and mitigation. We demonstrate its utility in two applications: (1) benchmarking image quality metrics and developing Semantic Hallucination Assessment via Feature Evaluation (SHAFE), a feature-based metric with soft-attention pooling that improves hallucination sensitivity over traditional metrics; and (2) training reference-free hallucination detectors that generalize to real restoration failures. Together, HalluGen and its open dataset establish the first scalable foundation for evaluating hallucinations in safety-critical image restoration.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 3 Dec 2025.
Builds on
35 of 51 references.
References · 35
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2023Segment Anythingcited by 8,158
- 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2017Enhanced Deep Residual Networks for Single Image Super-Resolutioncited by 1,543
- 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
- 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,250
- 2022Denoising Diffusion Restoration Modelscited by 951
- 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 792
- 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
- 2017The Perception-Distortion Tradeoffcited by 582
23 more
- 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 574
- 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
- 2019Making Convolutional Networks Shift-Invariant Againcited by 390
- 2017DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networkscited by 366
- 2023Denoising Diffusion Models for Plug-and-Play Image Restorationcited by 335
- 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
- 2024Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealingcited by 157
- 2022Solving 3D Inverse Problems using Pre-trained 2D Diffusion Modelscited by 130
- 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
- 2025HalluLens: LLM Hallucination Benchmarkcited by 88
- 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 87
- 2023Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusioncited by 78
- 2022Multiscale Structure Guided Diffusion for Image Deblurringcited by 76
- 2020On hallucinations in tomographic image reconstructioncited by 66
- 201711K Hands: Gender recognition and biometric identification using a large dataset of hand imagescited by 46
- 2015ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairscited by 35
- 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 30
- 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27
- 2024Hallucination Index: An Image Quality Metric for Generative Reconstruction Modelscited by 25
- 2024Tackling Structural Hallucination in Image Translation with Local Diffusioncited by 21
- 2023Low-field magnetic resonance image enhancement via stochastic image quality transfercited by 19
- 2023A 3D Conditional Diffusion Model for Image Quality Transfer -- An Application to Low-Field MRIcited by 13
- 2025Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulationcited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.