# HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion

arXiv: https://arxiv.org/abs/2510.03122 · HTML: https://mlsanity.com/paper/2510.03122/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models](https://mlsanity.com/paper/2304.12526/) (2023)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI](https://mlsanity.com/paper/1907.02431/) (2019)
- [MindBridge: A Cross-Subject Brain Decoding Framework](https://mlsanity.com/paper/2404.07850/) (2024)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion](https://mlsanity.com/paper/2308.04249/) (2023)
- [Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN](https://mlsanity.com/paper/2001.11761/) (2020)
- [DREAM: Visual Decoding from Reversing Human Visual System](https://mlsanity.com/paper/2310.02265/) (2023)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity](https://mlsanity.com/paper/2308.07428/) (2023)
- [NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation](https://mlsanity.com/paper/2403.18211/) (2024)
- [Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN](https://mlsanity.com/paper/2101.12083/) (2021)
- [Stable Diffusion is Unstable](https://mlsanity.com/paper/2306.02583/) (2023)
- [Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity](https://mlsanity.com/paper/2403.20022/) (2024)
- [BrainGuard: Privacy-Preserving Multisubject Image Reconstructions from Brain Activities](https://mlsanity.com/paper/2501.14309/) (2025)

## Built on by (most cited first, 1 of 1)

- [HyNeuralMap: Hyperbolic Mapping of Visual Semantics to Neural Hierarchies](https://mlsanity.com/paper/2605.09392/) (2026)

## Abstract (arXiv)

The reconstruction of visual information from brain activity fosters interdisciplinary integration between neuroscience and computer vision. However, existing methods still face challenges in accurately recovering highly complex visual stimuli. This difficulty stems from the characteristics of natural scenes: low-level features exhibit heterogeneity, while high-level features show semantic entanglement due to contextual overlaps. Inspired by the hierarchical representation theory of the visual cortex, we propose the HAVIR model, which separates the visual cortex into two hierarchical regions and extracts distinct features from each. Specifically, the Structural Generator extracts structural information from spatial processing voxels and converts it into latent diffusion priors, while the Semantic Extractor converts semantic processing voxels into CLIP embeddings. These components are integrated via the Versatile Diffusion model to synthesize the final image. Experimental results demonstrate that HAVIR enhances both the structural and semantic quality of reconstructions, even in complex scenes, and outperforms existing models.