# Pixels2Points: Fusing 2D and 3D Features for Facial Skin Segmentation

arXiv: https://arxiv.org/abs/2504.19718 · HTML: https://mlsanity.com/paper/2504.19718/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 13 of 13)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)
- [Probing the 3D Awareness of Visual Foundation Models](https://mlsanity.com/paper/2404.08636/) (2024)
- [DiffusionNet: Discretization Agnostic Learning on Surfaces](https://mlsanity.com/paper/2012.00888/) (2020)
- [Virtual Multi-view Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2007.13138/) (2020)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [ODIN: A Single Model for 2D and 3D Segmentation](https://mlsanity.com/paper/2401.02416/) (2024)
- [GarNet++: Improving Fast and Accurate Static3D Cloth Draping by Curvature Loss](https://mlsanity.com/paper/2007.10867/) (2020)
- [Parameter Efficient Local Implicit Image Function Network for Face Segmentation](https://mlsanity.com/paper/2303.15122/) (2023)
- [CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement](https://mlsanity.com/paper/2404.02225/) (2024)

## Built on by (most cited first, 2 of 2)

- [Representing 3D Faces with Learnable B-Spline Volumes](https://mlsanity.com/paper/2604.12894/) (2026)
- [FaceParts: Segmentation and Editing of Gaussian Splatting](https://mlsanity.com/paper/2605.13853/) (2026)

## Abstract (arXiv)

Face registration deforms a template mesh to closely fit a 3D face scan, the quality of which commonly degrades in non-skin regions (e.g., hair, beard, accessories), because the optimized template-to-scan distance pulls the template mesh towards the noisy scan surface. Improving registration quality requires a clean separation of skin and non-skin regions on the scan mesh. Existing image-based (2D) or scan-based (3D) segmentation methods however perform poorly. Image-based segmentation outputs multi-view inconsistent masks, and they cannot account for scan inaccuracies or scan-image misalignment, while scan-based methods suffer from lower spatial resolution compared to images. In this work, we introduce a novel method that accurately separates skin from non-skin geometry on 3D human head scans. For this, our method extracts features from multi-view images using a frozen image foundation model and aggregates these features in 3D. These lifted 2D features are then fused with 3D geometric features extracted from the scan mesh, to then predict a segmentation mask directly on the scan mesh. We show that our segmentations improve the registration accuracy over pure 2D or 3D segmentation methods by 8.89% and 14.3%, respectively. Although trained only on synthetic data, our model generalizes well to real data.