# Controllable Mind Visual Diffusion Model

arXiv: https://arxiv.org/abs/2305.10135 · HTML: https://mlsanity.com/paper/2305.10135/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 17 May 2023.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation](https://mlsanity.com/paper/2211.09869/) (2022)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Diffusion-SDF: Text-to-Shape via Voxelized Diffusion](https://mlsanity.com/paper/2212.03293/) (2022)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)

## Built on by (most cited first, 30 of 30)

- [Brain decoding: toward real-time reconstruction of visual perception](https://mlsanity.com/paper/2310.19812/) (2023)
- [Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Tasks](https://mlsanity.com/paper/2501.15891/) (2025)
- [ZONE: Zero-Shot Instruction-Guided Local Editing](https://mlsanity.com/paper/2312.16794/) (2023)
- [DREAM: Visual Decoding from Reversing Human Visual System](https://mlsanity.com/paper/2310.02265/) (2023)
- [EditWorld: Simulating World Dynamics for Instruction-Following Image Editing](https://mlsanity.com/paper/2405.14785/) (2024)
- [NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation](https://mlsanity.com/paper/2403.18211/) (2024)
- [Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs](https://mlsanity.com/paper/2306.11536/) (2023)
- [Scaling laws for decoding images from brain activity](https://mlsanity.com/paper/2501.15322/) (2025)
- [Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesis](https://mlsanity.com/paper/2410.07155/) (2024)
- [Seeing through the Brain: Image Reconstruction of Visual Perception from Human Brain Signals](https://mlsanity.com/paper/2308.02510/) (2023)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models](https://mlsanity.com/paper/2410.02780/) (2024)
- [DiffuX2CT: Diffusion Learning to Reconstruct CT Images from Biplanar X-Rays](https://mlsanity.com/paper/2407.13545/) (2024)
- [Naturalistic Music Decoding from EEG Data via Latent Diffusion Models](https://mlsanity.com/paper/2405.09062/) (2024)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding](https://mlsanity.com/paper/2510.27128/) (2025)
- [A Survey of fMRI to Image Reconstruction](https://mlsanity.com/paper/2502.16861/) (2025)
- [See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI](https://mlsanity.com/paper/2403.06361/) (2024)
- [Reanimating Images using Neural Representations of Dynamic Stimuli](https://mlsanity.com/paper/2406.02659/) (2024)
- [Towards Adversarial Robustness via Debiased High-Confidence Logit Alignment](https://mlsanity.com/paper/2408.06079/) (2024)
- [BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance](https://mlsanity.com/paper/2409.14021/) (2024)
- [Neural-Driven Image Editing](https://mlsanity.com/paper/2507.05397/) (2025)
- [Cross-Subject Mind Decoding from Inaccurate Representations](https://mlsanity.com/paper/2507.19071/) (2025)
- [Towards Interpretable Visual Decoding with Attention to Brain Representations](https://mlsanity.com/paper/2509.23566/) (2025)
- [Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding](https://mlsanity.com/paper/2603.07625/) (2026)
- [SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding](https://mlsanity.com/paper/2503.06437/) (2025)
- [Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction](https://mlsanity.com/paper/2510.22335/) (2025)
- [The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling](https://mlsanity.com/paper/2601.15071/) (2026)
- [MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment](https://mlsanity.com/paper/2607.00410/) (2026)
- [Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability](https://mlsanity.com/paper/2610.13763/) (2026)

## Abstract (arXiv)

Brain signal visualization has emerged as an active research area, serving as a critical interface between the human visual system and computer vision models. Although diffusion models have shown promise in analyzing functional magnetic resonance imaging (fMRI) data, including reconstructing high-quality images consistent with original visual stimuli, their accuracy in extracting semantic and silhouette information from brain signals remains limited. In this regard, we propose a novel approach, referred to as Controllable Mind Visual Diffusion Model (CMVDM). CMVDM extracts semantic and silhouette information from fMRI data using attribute alignment and assistant networks. Additionally, a residual block is incorporated to capture information beyond semantic and silhouette features. We then leverage a control model to fully exploit the extracted information for image synthesis, resulting in generated images that closely resemble the visual stimuli in terms of semantics and silhouette. Through extensive experimentation, we demonstrate that CMVDM outperforms existing state-of-the-art methods both qualitatively and quantitatively.