# Double-Flow GAN model for the reconstruction of perceived faces from brain activities

arXiv: https://arxiv.org/abs/2312.07478 · HTML: https://mlsanity.com/paper/2312.07478/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding](https://mlsanity.com/paper/2211.06956/) (2022)
- [Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation](https://mlsanity.com/paper/2104.08771/) (2021)
- [From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI](https://mlsanity.com/paper/1907.02431/) (2019)
- [Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN](https://mlsanity.com/paper/2001.11761/) (2020)
- [Reconstructing Faces from fMRI Patterns using Deep Generative Neural Networks](https://mlsanity.com/paper/1810.03856/) (2018)
- [DCNN-GAN: Reconstructing Realistic Image from fMRI](https://mlsanity.com/paper/1901.07368/) (2019)
- [Sharing deep generative representation for perceived image reconstruction from human brain activity](https://mlsanity.com/paper/1704.07575/) (2017)

## Abstract (arXiv)

Face plays an important role in humans visual perception, and reconstructing perceived faces from brain activities is challenging because of its difficulty in extracting high-level features and maintaining consistency of multiple face attributes, such as expression, identity, gender, etc. In this study, we proposed a novel reconstruction framework, which we called Double-Flow GAN, that can enhance the capability of discriminator and handle imbalances in images from certain domains that are too easy for generators. We also designed a pretraining process that uses features extracted from images as conditions for making it possible to pretrain the conditional reconstruction model from fMRI in a larger pure image dataset. Moreover, we developed a simple pretrained model for fMRI alignment to alleviate the problem of cross-subject reconstruction due to the variations of brain structure among different subjects. We conducted experiments by using our proposed method and traditional reconstruction models. Results showed that the proposed method is significant at accurately reconstructing multiple face attributes, outperforms the previous reconstruction models, and exhibited state-of-the-art reconstruction abilities.