Rethinking Voice-Face Correlation: A Geometry View
Not on the map. Why?
Abstract (arXiv)
Previous works on voice-face matching and voice-guided face synthesis demonstrate strong correlations between voice and face, but mainly rely on coarse semantic cues such as gender, age, and emotion. In this paper, we aim to investigate the capability of reconstructing the 3D facial shape from voice from a geometry perspective without any semantic information. We propose a voice-anthropometric measurement (AM)-face paradigm, which identifies predictable facial AMs from the voice and uses them to guide 3D face reconstruction. By leveraging AMs as a proxy to link the voice and face geometry, we can eliminate the influence of unpredictable AMs and make the face geometry tractable. Our approach is evaluated on our proposed dataset with ground-truth 3D face scans and corresponding voice recordings, and we find significant correlations between voice and specific parts of the face geometry, such as the nasal cavity and cranium. Our work offers a new perspective on voice-face correlation and can serve as a good empirical study for anthropometry science.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 56 references that are in the map. First posted on arXiv 26 Jul 2023.
Builds on
22 of 56 references.
References · 22
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2022Generative Adversarial Networkscited by 5,627
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
- 2014Conditional Generative Adversarial Netscited by 2,162
- 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
- 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
- 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
- 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
- 2018Speaker Recognition from Raw Waveform with SincNetcited by 166
10 more
- 2018Lip Movements Generation at a Glancecited by 155
- 2019Speech2Face: Learning the Face Behind a Voicecited by 55
- 2020Cross-modal Speaker Verification and Recognition: A Multilingual Perspectivecited by 26
- 2020Audio-visual Speaker Recognition with a Cross-modal Discriminative Networkcited by 24
- 2021A Multi-View Approach To Audio-Visual Speaker Verificationcited by 23
- 2021Exploring Automatic COVID-19 Diagnosis via voice and symptoms from Crowdsourced Datacited by 21
- 2021Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Associationcited by 21
- 2019Attention-Augmented End-to-End Multi-Task Learning for Emotion Prediction from Speechcited by 17
- 2022Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?cited by 8
- 2023PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancementcited by 2
Built on by
Citing papers per year: 2023: 2; 2024: 0; 2025: 1.
3 papers.