# HAIFAI: Human-AI Interaction for Mental Face Reconstruction

arXiv: https://arxiv.org/abs/2412.06323 · HTML: https://mlsanity.com/paper/2412.06323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 50 of its 102 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation](https://mlsanity.com/paper/2011.12799/) (2020)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [AttGAN: Facial Attribute Editing by Only Changing What You Want](https://mlsanity.com/paper/1711.10678/) (2017)
- [Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning](https://mlsanity.com/paper/2004.11660/) (2020)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors](https://mlsanity.com/paper/2305.18274/) (2023)
- [EditGAN: High-Precision Semantic Image Editing](https://mlsanity.com/paper/2111.03186/) (2021)
- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [Natural scene reconstruction from fMRI signals using generative latent diffusion](https://mlsanity.com/paper/2303.05334/) (2023)
- [From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI](https://mlsanity.com/paper/1907.02431/) (2019)
- [A Latent Transformer for Disentangled Face Editing in Images and Videos](https://mlsanity.com/paper/2106.11895/) (2021)
- [Mask-Guided Portrait Editing with Conditional GANs](https://mlsanity.com/paper/1905.10346/) (2019)
- [CONFIG: Controllable Neural Face Image Generation](https://mlsanity.com/paper/2005.02671/) (2020)

## Abstract (arXiv)

We present HAIFAI - a novel two-stage system where humans and AI interact to tackle the challenging task of reconstructing a visual representation of a face that exists only in a person's mind. In the first stage, users iteratively rank images our reconstruction system presents based on their resemblance to a mental image. These rankings, in turn, allow the system to extract relevant image features, fuse them into a unified feature vector, and use a generative model to produce an initial reconstruction of the mental image. The second stage leverages an existing face editing method, allowing users to manually refine and further improve this reconstruction using an easy-to-use slider interface for face shape manipulation. To avoid the need for tedious human data collection for training the reconstruction system, we introduce a computational user model of human ranking behaviour. For this, we collected a small face ranking dataset through an online crowd-sourcing study containing data from 275 participants. We evaluate HAIFAI and an ablated version in a 12-participant user study and demonstrate that our approach outperforms the previous state of the art regarding reconstruction quality, usability, perceived workload, and reconstruction speed. We further validate the reconstructions in a subsequent face ranking study with 18 participants and show that HAIFAI achieves a new state-of-the-art identification rate of 60.6%. These findings represent a significant advancement towards developing new interactive intelligent systems capable of reliably and effortlessly reconstructing a user's mental image.