mlsanityData to 7 Oct 2026

Tackling Structural Hallucination in Image Translation with Local Diffusion

arXiv 2404.05980 ↗First posted 9 Apr 2024cs.CV cs.AI cs.LGcited by 21

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing "image hallucination" and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a "branching" module generates locally both within and outside OOD regions, and a "fusion" module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 33 of its 42 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

33 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A 3D Conditional Diffusion Model for Image Quality Transfer -- An Application to Low-Field MRI11 Nov

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2016Wide Residual Networkscited by 3,057
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
20 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2023Consistency Modelscited by 1,778
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  5. 2021Variational Diffusion Modelscited by 1,145
  6. 2017Deep Image Priorcited by 1,097
  7. 2022Denoising Diffusion Restoration Modelscited by 951
  8. 2021Towards Total Recall in Industrial Anomaly Detectioncited by 772
  9. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 574
  10. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  11. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  12. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  13. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  14. 2021Robust Compressed Sensing MRI with Deep Generative Priorscited by 284
  15. 2021FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flowscited by 225
  16. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  17. 2022Diffusion-GAN: Training GANs with Diffusioncited by 203
  18. 2022CFA: Coupled-hypersphere-based Feature Adaptation for Target-Oriented Anomaly Localizationcited by 135
  19. 2021On the Out-of-distribution Generalization of Probabilistic Image Modellingcited by 32
  20. 2023Distribution Shift Inversion for Out-of-Distribution Predictioncited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 6.

21 papers.

Sort:

By area:

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  2. 2024TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Modelscited by 12
  3. 2025Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulationcited by 11
  4. 2025Tackling Hallucination from Conditional Models for Medical Image Reconstruction with DynamicDPScited by 10
  5. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 5
  6. 2025CIVET: Systematic Evaluation of Understanding in VLMscited by 4
  7. 2025RODS: Robust Optimization Inspired Diffusion Sampling for Detecting and Reducing Hallucination in Generative Modelscited by 3
  8. 2025Responsible Diffusion: A Comprehensive Survey on Safety, Ethics, and Trust in Diffusion Modelscited by 3
  9. 2024Conditioning 3D Diffusion Models with 2D Images: Towards Standardized OCT Volumes through En Face-Informed Super-Resolutioncited by 1
  10. 2025On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Modelscited by 1
  11. 2025Unsupervised Multi-View Visual Anomaly Detection via Progressive Homography-Guided Alignmentcited by 1
  12. 2025HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restorationcited by 1
9 more
  1. 2026Local Intrinsic Dimension Unveils Hallucinations in Diffusion Modelscited by 1
  2. 2025Efficient and robust 3D blind harmonization for large domain gapscited by 0
  3. 2025IMPLICITSTAINER: Resolution Agnostic Data-Efficient Virtual Staining Using Neural Implicit Functionscited by 0
  4. 2025pyMEAL: A Multi-Encoder Augmentation-Aware-Learning Toolbox for Robust Medical Image Translationcited by 0
  5. 2025Visually grounded emotion regulation via diffusion models and user-driven reappraisalcited by 0
  6. 2026MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstructioncited by 0
  7. 2026Using Laplace Transform To Optimize the Hallucination of Generation Modelscited by 0
  8. 2026SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignmentcited by 0
  9. 2026Score-Control for Hallucination Reduction in Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hallucination Early Detection in Diffusion Models
  2. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolation
  3. 2025Mitigating Diffusion Model Hallucinations with Dynamic Guidance
  4. 2024Hallucination Index: An Image Quality Metric for Generative Reconstruction Models
  5. 2026HAD: Hallucination-Aware Diffusion Priors for 3D Reconstruction
  6. 2025Counting Hallucinations in Diffusion Models
  7. 2025Laplacian Score Sharpening for Mitigating Hallucination in Diffusion Models
  8. 2026MEND: Label-Free Detection, Localisation, and Correction of Latent Hallucination in World Models