mlsanityData to 14 Oct 2026

OFER: Occluded Face Expression Reconstruction

arXiv 2410.21629 ↗First posted 29 Oct 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Reconstructing 3D face models from a single image is an inherently ill-posed problem, which becomes even more challenging in the presence of occlusions. In addition to fewer available observations, occlusions introduce an extra source of ambiguity where multiple reconstructions can be equally valid. Despite the ubiquity of the problem, very few methods address its multi-hypothesis nature. In this paper we introduce OFER, a novel approach for single-image 3D face reconstruction that can generate plausible, diverse, and expressive 3D faces, even under strong occlusions. Specifically, we train two diffusion models to generate the shape and expression coefficients of a face parametric model, conditioned on the input image. This approach captures the multi-modal nature of the problem, generating a distribution of solutions as output. However, to maintain consistency across diverse expressions, the challenge is to select the best matching shape. To achieve this, we propose a novel ranking mechanism that sorts the outputs of the shape diffusion network based on predicted shape accuracy scores. We evaluate our method using standard benchmarks and introduce CO-545, a new protocol and dataset designed to assess the accuracy of expressive faces under occlusion. Our results show improved performance over occlusion-based methods, while also enabling the generation of diverse expressions for a given image.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 68 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

35 of 68 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  7. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 477
  8. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  9. 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 268
  10. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
  11. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  12. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
23 more
  1. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  2. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  3. 2022Towards Metrical Reconstruction of Human Facescited by 166
  4. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  5. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  6. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  7. 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 84
  8. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  9. 2023Generating images of rare concepts using pre-trained diffusion modelscited by 79
  10. 20223D face reconstruction with dense landmarkscited by 70
  11. 2017Adversarial Ranking for Language Generationcited by 50
  12. 2021Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracingcited by 48
  13. 2017End-to-end 3D face reconstruction with deep neural networkscited by 47
  14. 2020Learning-to-Rank with BERT in TF-Rankingcited by 36
  15. 2023Instant Multi-View Head Capture through Learnable Registrationcited by 34
  16. 2018DenseReg: Fully Convolutional Dense Shape Regression In-the-Wildcited by 32
  17. 2021SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstructioncited by 27
  18. 2018Evaluation of Dense 3D Reconstruction from 2D Face Images in the Wildcited by 22
  19. 2021Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentationcited by 20
  20. 2021On Recognizing Occluded Faces in the Wildcited by 15
  21. 2022Delving into High-Quality Synthetic Face Occlusion Segmentation Datasetscited by 11
  22. 2021Learning Free-Form Deformation for 3D Face Reconstruction from In-The-Wild Imagescited by 7
  23. 2021Generating Diverse 3D Reconstructions from a Single Occluded Face Imagecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MoDE: Mixture of Diffusion Experts for Any Occluded Face Recognition
  2. 2023O2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model
  3. 2023Fake It Without Making It: Conditioned Face Generation for Accurate 3D Face Reconstruction
  4. 2024FaceLift: Learning Generalizable Single Image 3D Face Reconstruction from Synthetic Heads
  5. 2023FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models
  6. 2026FACE: A Face-based Autoregressive Representation for High-Fidelity and Efficient Mesh Generation
  7. 2026Face inpainting with Identity Preserving Latent Diffusion Models
  8. 2026Splatshot: 3D Face Avatar Generation from a Single Unconstrained Photo