Cross-modal Face- and Voice-style Transfer
Not on the map. Why?
Abstract (arXiv)
Image-to-image translation and voice conversion enable the generation of a new facial image and voice while maintaining some of the semantics such as a pose in an image and linguistic content in audio, respectively. They can aid in the content-creation process in many applications. However, as they are limited to the conversion within each modality, matching the impression of the generated face and voice remains an open question. We propose a cross-modal style transfer framework called XFaVoT that jointly learns four tasks: image translation and voice conversion tasks with audio or image guidance, which enables the generation of ``face that matches given voice" and ``voice that matches given face", and intra-modality translation tasks with a single framework. Experimental results on multiple datasets show that XFaVoT achieves cross-modal style translation of image and voice, outperforming baselines in terms of quality, diversity, and face-voice correspondence.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 61 of its 82 references that are in the map. First posted on arXiv 27 Feb 2023.
Builds on
61 of 82 references.
References · 61
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
- 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
- 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
- 2016Identity Mappings in Deep Residual Networkscited by 1,960
- 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
- 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
38 more
- 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,095
- 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
- 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
- 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
- 2018Which Training Methods for GANs do actually Converge?cited by 672
- 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
- 2017Unsupervised Image-to-Image Translation Networkscited by 602
- 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
- 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
- 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
- 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
- 2017Toward Multimodal Image-to-Image Translationcited by 295
- 2020Self-Supervised MultiModal Versatile Networkscited by 276
- 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
- 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
- 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
- 2019ManiGAN: Text-Guided Image Manipulationcited by 156
- 2019Mode Seeking Generative Adversarial Networks for Diverse Image Synthesiscited by 138
- 2019Adaptive Wing Loss for Robust Face Alignment via Heatmap Regressioncited by 127
- 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
- 2017Semantic Image Synthesis via Adversarial Learningcited by 87
- 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
- 2019Diversity-Sensitive Conditional Generative Adversarial Networkscited by 77
- 2018Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Datacited by 76
- 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
- 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
- 2021The Spatially-Correlative Loss for Various Image Translation Taskscited by 66
- 2021StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversioncited by 57
- 2019Speech2Face: Learning the Face Behind a Voicecited by 55
- 2018Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformationcited by 49
- 2021JoJoGAN: One Shot Face Stylizationcited by 49
- 2021Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translationcited by 45
- 2020AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalizationcited by 44
- 2022Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Taskscited by 42
- 2018Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistencycited by 41
- 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
- 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
- 2021Deep Learning Based Assessment of Synthetic Speech Naturalnesscited by 35
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.