mlsanityData to 7 Oct 2026

Hear Your Face: Face-based voice conversion with F0 estimation

arXiv 2408.09802 ↗First posted 19 Aug 2024cs.SD cs.CV eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

This paper delves into the emerging field of face-based voice conversion, leveraging the unique relationship between an individual's facial features and their vocal characteristics. We present a novel face-based voice conversion framework that particularly utilizes the average fundamental frequency of the target speaker, derived solely from their facial images. Through extensive analysis, our framework demonstrates superior speech generation quality and the ability to align facial features with voice characteristics, including tracking of the target speaker's fundamental frequency.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

23 of 32 references.

References · 23

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2016Density estimation using Real NVPcited by 2,029
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2018Self-Attention with Relative Position Representationscited by 1,112
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  7. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  8. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  9. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  10. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  11. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  12. 2019Speech2Face: Learning the Face Behind a Voicecited by 55
11 more
  1. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  2. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  3. 2023Imaginary Voice: Face-styled Diffusion Model for Text-to-Speechcited by 27
  4. 2018On Learning Associations of Faces and Voicescited by 26
  5. 2018Neural source-filter-based waveform model for statistical parametric speech synthesiscited by 21
  6. 2021Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Associationcited by 21
  7. 2020From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speechcited by 17
  8. 2019Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversioncited by 12
  9. 2022Residual-guided Personalized Speech Synthesis based on Face Imagecited by 10
  10. 2023Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignmentcited by 8
  11. 2023Cross-modal Face- and Voice-style Transfercited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
  2. 2024Audio-based Kinship Verification Using Age Domain Conversioncited by 3
  3. 2025MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakerscited by 2
  4. 2025MuteSwap: Visual-informed Silent Video Identity Conversioncited by 0