# Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration

arXiv: https://arxiv.org/abs/2403.02629 · HTML: https://mlsanity.com/paper/2403.02629/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 63 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [I M Avatar: Implicit Morphable Head Avatars from Videos](https://mlsanity.com/paper/2112.07471/) (2021)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [RigNeRF: Fully Controllable Neural 3D Portraits](https://mlsanity.com/paper/2206.06481/) (2022)
- [Learning Neural Parametric Head Models](https://mlsanity.com/paper/2212.02761/) (2022)
- [Towards a complete 3D morphable model of the human head](https://mlsanity.com/paper/1911.08008/) (2019)
- [A Survey of Non-Rigid 3D Registration](https://mlsanity.com/paper/2203.07858/) (2022)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [Shape Preserving Facial Landmarks with Graph Attention Networks](https://mlsanity.com/paper/2210.07233/) (2022)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [Neural Parameterization for Dynamic Human Head Editing](https://mlsanity.com/paper/2207.00210/) (2022)
- [Dense 3D Face Correspondence](https://mlsanity.com/paper/1410.5058/) (2014)
- [Mesh Density Adaptation for Template-based Shape Reconstruction](https://mlsanity.com/paper/2307.16205/) (2023)
- [Laplacian ICP for Progressive Registration of 3D Human Head Meshes](https://mlsanity.com/paper/2302.02194/) (2023)

## Abstract (arXiv)

This paper presents a Geometric-Photometric Joint Alignment~(GPJA) method, which aligns discrete human expressions at pixel-level accuracy by combining geometric and photometric information. Common practices for registering human heads typically involve aligning landmarks with facial template meshes using geometry processing approaches, but often overlook dense pixel-level photometric consistency. This oversight leads to inconsistent texture parametrization across different expressions, hindering the creation of topologically consistent head meshes widely used in movies and games. GPJA overcomes this limitation by leveraging differentiable rendering to align vertices with target expressions, achieving joint alignment in both geometry and photometric appearances automatically, without requiring semantic annotation or pre-aligned meshes for training. It features a holistic rendering alignment mechanism and a multiscale regularized optimization for robust convergence on large deformation. The method utilizes derivatives at vertex positions for supervision and employs a gradient-based algorithm which guarantees smoothness and avoids topological artifacts during the geometry evolution. Experimental results demonstrate faithful alignment under various expressions, surpassing the conventional non-rigid ICP-based methods and the state-of-the-art deep learning based method. In practical, our method generates meshes of the same subject across diverse expressions, all with the same texture parametrization. This consistency benefits face animation, re-parametrization, and other batch operations for face modeling and applications with enhanced efficiency.