mlsanityData to 7 Oct 2026

Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry

arXiv 2110.09772 ↗First posted 19 Oct 2021cs.CV cs.GRcited by 27

Not on the map. Why?

Abstract (arXiv)

This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks to predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling. Our synergy process leverages a representation cycle for 3DMM parameters and 3D landmarks. 3D landmarks can be extracted and refined from face meshes built by 3DMM parameters. We next reverse the representation direction and show that predicting 3DMM parameters from sparse 3D landmarks improves the information flow. Together we create a synergy process that utilizes the relation between 3D landmarks and 3DMM parameters, and they collaboratively contribute to better performance. We extensively validate our contribution on full tasks of facial geometry prediction and show our superior and robust performance on these tasks for various scenarios. Particularly, we adopt only simple and widely-used network operations to attain fast and accurate facial geometry prediction. Codes and data: https://choyingw.github.io/works/SynergyNet/

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 44 of its 78 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

44 of 78 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  7. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
  8. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  9. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  10. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  11. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  12. 2020ResNeSt: Split-Attention Networkscited by 445
32 more
  1. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  2. 2018Deep Video Portraitscited by 265
  3. 2020Towards Fast, Accurate and Stable 3D Dense Face Alignmentcited by 219
  4. 2018Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Networkcited by 195
  5. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  6. 2017Fine-Grained Head Pose Estimation Without Keypointscited by 163
  7. 2018Face Alignment in Full Pose Range: A 3D Total Solutioncited by 147
  8. 2018Look at Boundary: A Boundary-Aware Face Alignment Algorithmcited by 137
  9. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  10. 2020Rethinking Pseudo-LiDAR Representationcited by 109
  11. 2017Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networkscited by 108
  12. 2019Grid-GCN for Fast and Scalable Point Cloud Learningcited by 103
  13. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  14. 2018FML: Face Model Learning from Videoscited by 78
  15. 2020Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistencycited by 78
  16. 2017Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regressioncited by 66
  17. 2018Style Aggregated Network for Facial Landmark Detectioncited by 60
  18. 2019MVF-Net: Multi-View 3D Face Morphable Model Regressioncited by 45
  19. 2021Masked Face Recognition Challenge: The InsightFace Track Reportcited by 41
  20. 2020A Vector-based Representation to Enhance Head Pose Estimationcited by 34
  21. 2019Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detectioncited by 28
  22. 2017Dense Face Alignmentcited by 24
  23. 2019Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completioncited by 20
  24. 2017Faster Than Real-time Facial Alignment: A 3D Spatial Transformer Network Approach in Unconstrained Posescited by 19
  25. 2021Self-Supervised 3D Face Reconstruction via Conditional Estimationcited by 17
  26. 2020RankPose: Learning Generalised Feature with Rank Supervision for Head Pose Estimationcited by 10
  27. 2020Scene Completeness-Aware Lidar Depth Completion for Driving Scenariocited by 7
  28. 2017Learning Dense Facial Correspondences in Unconstrained Imagescited by 6
  29. 2019Occluded Face Recognition Using Low-rank Regression with Generalized Gradient Directioncited by 6
  30. 2016A Comprehensive Performance Evaluation of Deformable Face Tracking "In-the-Wild"cited by 5
  31. 2020Geometry-Aware Instance Segmentation with Disparity Mapscited by 4
  32. 2018Efficient Multi-Domain Dictionary Learning with GANscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 12; 2024: 7; 2025: 2; 2026: 2.

27 papers.

Sort:

By area:

  1. 2022DAD-3DHeads: A Large-scale Dense, Accurate and Diverse Dataset for 3D Head Alignment from a Single Imagecited by 31
  2. 2023HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Detailscited by 22
  3. 2024Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cuescited by 11
  4. 2023ImFace++: A Sophisticated Nonlinear 3D Morphable Face Model with Implicit Neural Representationscited by 10
  5. 2024DEGAS: Detailed Expressions on Full-Body Gaussian Avatarscited by 10
  6. 2022Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?cited by 8
  7. 2023DirectMHP: Direct 2D Multi-Person Head Pose Estimation with Full-range Anglescited by 6
  8. 2023DSFNet: Dual Space Fusion Network for Occlusion-Robust 3D Dense Face Alignmentcited by 6
  9. 2023A Perceptual Shape Loss for Monocular 3D Face Reconstructioncited by 6
  10. 2023EFHQ: Multi-purpose ExtremePose-Face-HQ datasetcited by 6
  11. 2024VGGHeads: 3D Multi Head Alignment with a Large-Scale Synthetic Datasetcited by 4
  12. 2022Joint Multi-Person Body Detection and Orientation Estimation via One Unified Embeddingcited by 3
15 more
  1. 2023Meta-Optimization for Higher Model Generalizability in Single-Image Depth Predictioncited by 3
  2. 2022FLNeRF: 3D Facial Landmarks Estimation in Neural Radiance Fieldscited by 2
  3. 2024Semi-Supervised Unconstrained Head Pose Estimation in the Wildcited by 2
  4. 2024FaceLift: Semi-supervised 3D Facial Landmark Localizationcited by 2
  5. 2023A Real-Time Multi-Task Learning System for Joint Detection of Face, Facial Landmark and Head Posecited by 1
  6. 2023InSpaceType: Reconsider Space Type in Indoor Monocular Depth Estimationcited by 1
  7. 2023A survey and classification of face alignment methods based on face modelscited by 1
  8. 20253D Face Reconstruction Error Decomposed: A Modular Benchmark for Fair and Fast Method Evaluationcited by 1
  9. 20233D Face Alignment Through Fusion of Head Pose Information and Featurescited by 0
  10. 2023Student Classroom Behavior Detection based on Spatio-Temporal Network and Multi-Model Fusioncited by 0
  11. 2024On the power of data augmentation for head pose estimationcited by 0
  12. 2024Bidirectional Regression for Monocular 6DoF Head Pose Estimation and Reference System Alignmentcited by 0
  13. 2025Geometry-Aware Video Inpainting for Joint Headset Occlusion Removal and Face Reconstruction in Social XRcited by 0
  14. 2026Profile-Specific 3DMM Regression from a Single Lateral Face Imagecited by 0
  15. 2026FADRA: Frequency-Aware Diffusion with Residual Adaptation for Video Face Restorationcited by 0