# Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity

arXiv: https://arxiv.org/abs/2403.20022 · HTML: https://mlsanity.com/paper/2403.20022/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical MoE, one of 12 directions in Mixture-of-experts.
- Builds on 38 of its 76 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Placed in

- Mixture-of-experts / [Medical and biomedical MoE](https://mlsanity.com/moe/medical-and-biomedical-moe/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis](https://mlsanity.com/paper/2402.05408/) (2024)
- [Generic decoding of seen and imagined objects using hierarchical visual features](https://mlsanity.com/paper/1510.06479/) (2015)
- [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](https://mlsanity.com/paper/2106.03760/) (2021)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [Mind Reader: Reconstructing complex images from brain activities](https://mlsanity.com/paper/2210.01769/) (2022)
- [Bird's-Eye-View Scene Graph for Vision-Language Navigation](https://mlsanity.com/paper/2308.04758/) (2023)
- [From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI](https://mlsanity.com/paper/1907.02431/) (2019)
- [BOLD5000: A public fMRI dataset of 5000 images](https://mlsanity.com/paper/1809.01281/) (2018)
- [DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)](https://mlsanity.com/paper/2401.08392/) (2024)
- [Network of Experts for Large-Scale Image Categorization](https://mlsanity.com/paper/1604.06119/) (2016)
- [Decoding natural image stimuli from fMRI data with a surface-based convolutional network](https://mlsanity.com/paper/2212.02409/) (2022)
- [MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion](https://mlsanity.com/paper/2308.04249/) (2023)
- [TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts](https://mlsanity.com/paper/2307.15324/) (2023)

## Built on by (most cited first, 22 of 22)

- [MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis](https://mlsanity.com/paper/2407.02329/) (2024)
- [Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior](https://mlsanity.com/paper/2503.04207/) (2025)
- [MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI](https://mlsanity.com/paper/2503.02351/) (2025)
- [InfiniDreamer: Arbitrarily Long Human Motion Generation via Segment Score Distillation](https://mlsanity.com/paper/2411.18303/) (2024)
- [Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2411.13378/) (2024)
- [A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli](https://mlsanity.com/paper/2503.15978/) (2025)
- [Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI](https://mlsanity.com/paper/2505.14556/) (2025)
- [COBRA: A Continual Learning Approach to Vision-Brain Understanding](https://mlsanity.com/paper/2411.17475/) (2024)
- [Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing](https://mlsanity.com/paper/2503.10663/) (2025)
- [From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structure](https://mlsanity.com/paper/2507.16389/) (2025)
- [Cross-Subject Mind Decoding from Inaccurate Representations](https://mlsanity.com/paper/2507.19071/) (2025)
- [HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion](https://mlsanity.com/paper/2510.03122/) (2025)
- [SynMind: Reducing Semantic Hallucination in fMRI-Based Image Reconstruction](https://mlsanity.com/paper/2601.17857/) (2026)
- [SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance](https://mlsanity.com/paper/2602.21819/) (2026)
- [Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding](https://mlsanity.com/paper/2603.07625/) (2026)
- [Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction](https://mlsanity.com/paper/2605.14569/) (2026)
- [HAVIR: HierArchical Vision to Image Reconstruction using CLIP-Guided Versatile Diffusion](https://mlsanity.com/paper/2506.06035/) (2025)
- [Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI](https://mlsanity.com/paper/2510.16196/) (2025)
- [Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction](https://mlsanity.com/paper/2510.22335/) (2025)
- [Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping](https://mlsanity.com/paper/2511.11437/) (2025)
- [The Pictorial Cortex: Zero-Shot Cross-Subject fMRI-to-Image Reconstruction via Compositional Latent Modeling](https://mlsanity.com/paper/2601.15071/) (2026)
- [StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation](https://mlsanity.com/paper/2605.02586/) (2026)

## Abstract (arXiv)

Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface. The inherent variability in brain function between individuals leads existing literature to focus on acquiring separate models for each individual using their respective brain signal data, ignoring commonalities between these data. In this article, we devise Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) obtained from different subjects. Psychometry incorporates an omni mixture-of-experts (Omni MoE) module where all the experts work together to capture the inter-subject commonalities, while each expert associated with subject-specific parameters copes with the individual differences. Moreover, Psychometry is equipped with a retrieval-enhanced inference strategy, termed Ecphory, which aims to enhance the learned fMRI representation via retrieving from prestored subject-specific memories. These designs collectively render Psychometry omnifit and efficient, enabling it to capture both inter-subject commonality and individual specificity across subjects. As a result, the enhanced fMRI representations serve as conditional signals to guide a generation model to reconstruct high-quality and realistic images, establishing Psychometry as state-of-the-art in terms of both high-level and low-level metrics.