mlsanityData to 7 Oct 2026

Expression-preserving face frontalization improves visually assisted speech processing

arXiv 2204.02810 ↗First posted 6 Apr 2022cs.CV cs.SD eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a frontalization methodology that preserves non-rigid facial deformations in order to boost the performance of visually assisted speech communication. The method alternates between the estimation of (i)~the rigid transformation (scale, rotation, and translation) and (ii)~the non-rigid deformation between an arbitrarily-viewed face and a face model. The method has two important merits: it can deal with non-Gaussian errors in the data and it incorporates a dynamical face deformation model. For that purpose, we use the generalized Student t-distribution in combination with a linear dynamic system in order to account for both rigid head motions and time-varying facial deformations caused by speech production. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability of the method to preserve facial expressions. The method is thoroughly evaluated and compared with several state of the art methods, either based on traditional geometric models or on deep learning. Moreover, we show that the method, when incorporated into deep learning pipelines, namely lip reading and speech enhancement, improves word recognition and speech intelligibilty scores by a considerable margin. Supplemental material is accessible at https://team.inria.fr/robotlearn/research/facefrontalization/

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 74 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

25 of 74 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  4. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  5. 2018SDR - half-baked or well done?cited by 420
  6. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  7. 2018Face Alignment in Full Pose Range: A 3D Total Solutioncited by 147
  8. 2020Lipreading using Temporal Convolutional Networkscited by 98
  9. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  10. 2017Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesiscited by 75
  11. 2017Towards Large-Pose Face Frontalization in the Wildcited by 50
  12. 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 44
13 more
  1. 2020Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Imagescited by 37
  2. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  3. 2014Effective Face Frontalization in Unconstrained Imagescited by 34
  4. 2019A variance modeling framework based on variational autoencoders for speech enhancementcited by 33
  5. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 27
  6. 2020Lip-reading with Densely Connected Temporal Convolutional Networkscited by 26
  7. 2018Lip-Reading Driven Deep Learning Approach for Speech Enhancementcited by 14
  8. 2020Dual-Attention GAN for Large-Pose Face Frontalizationcited by 13
  9. 2019Towards Pose-invariant Lip-Readingcited by 12
  10. 2019Mixture of Inference Networks for VAE-based Audio-visual Speech Enhancementcited by 11
  11. 2016Two-stage Convolutional Part Heatmap Regression for the 1st 3D Face Alignment in the Wild (3DFAW) Challengecited by 8
  12. 2018GridFace: Face Rectification via Learning Local Homography Transformationscited by 6
  13. 2019Dual Attention MobDenseNet(DAMDNet) for Robust 3D Face Alignmentcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2023Breathing Life into Faces: Speech-driven 3D Facial Animation with Natural Head Pose and Detailed Shapecited by 3
  2. 2022Audio-visual speech enhancement with a deep Kalman filter generative modelcited by 2
  3. 2024eMotion-GAN: A Motion-based GAN for Photorealistic and Facial Expression Preserving Frontal View Synthesiscited by 0
  4. 2026Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognitioncited by 0