PaCo-FR: Patch-Pixel Aligned End-to-End Codebook Learning for Facial Representation Pre-training
Not on the map. Why?
Abstract (arXiv)
Facial representation pre-training is crucial for tasks like facial recognition, expression analysis, and virtual reality. However, existing methods face three key challenges: (1) failing to capture distinct facial features and fine-grained semantics, (2) ignoring the spatial structure inherent to facial anatomy, and (3) inefficiently utilizing limited labeled data. To overcome these, we introduce PaCo-FR, an unsupervised framework that combines masked image modeling with patch-pixel alignment. Our approach integrates three innovative components: (1) a structured masking strategy that preserves spatial coherence by aligning with semantically meaningful facial regions, (2) a novel patch-based codebook that enhances feature discrimination with multiple candidate tokens, and (3) spatial consistency constraints that preserve geometric relationships between facial components. PaCo-FR achieves state-of-the-art performance across several facial analysis tasks with just 2 million unlabeled images for pre-training. Our method demonstrates significant improvements, particularly in scenarios with varying poses, occlusions, and lighting conditions. We believe this work advances facial representation learning and offers a scalable, efficient solution that reduces reliance on expensive annotated datasets, driving more effective facial analysis systems.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 69 references that are in the map. First posted on arXiv 13 Aug 2025.
Builds on
47 of 69 references.
References · 47
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2017Neural Discrete Representation Learningcited by 4,712
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
35 more
- 2016Pyramid Scene Parsing Networkcited by 2,479
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2021Multiscale Vision Transformerscited by 897
- 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
- 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 268
- 2019High-Resolution Representations for Labeling Pixels and Regionscited by 211
- 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
- 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
- 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
- 2022Towards Metrical Reconstruction of Human Facescited by 166
- 2018Look at Boundary: A Boundary-Aware Face Alignment Algorithmcited by 138
- 2019Adaptive Wing Loss for Robust Face Alignment via Heatmap Regressioncited by 128
- 2017Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networkscited by 109
- 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
- 2020LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihoodcited by 87
- 2020Towards Universal Representation Learning for Deep Face Recognitioncited by 78
- 2018Style Aggregated Network for Facial Landmark Detectioncited by 60
- 2019DeCaFA: Deep Convolutional Cascade for Face Alignment In The Wildcited by 46
- 2017Deep Alignment Network: A convolutional neural network for robust face alignmentcited by 44
- 2021ADNet: Leveraging Error-Bias Towards Normal Direction in Face Alignmentcited by 34
- 2023Instant Multi-View Head Capture through Learnable Registrationcited by 34
- 2019Aggregation via Separation: Boosting Facial Landmark Detector with Semi-Supervised Style Translationcited by 33
- 2019Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detectioncited by 28
- 2020Edge-aware Graph Representation Learning and Reasoning for Face Parsingcited by 26
- 2021Pre-training strategies and datasets for facial representation learningcited by 26
- 2016Dynamic Attention-controlled Cascaded Shape Regression Exploiting Training Data Augmentation and Fuzzy-set Sample Weightingcited by 20
- 2023MagicAvatar: Multimodal Avatar Generation and Animationcited by 17
- 2024Self-Supervised Facial Representation Learning with Facial Region Awarenesscited by 17
- 2019Laplace Landmark Localizationcited by 16
- 2021AGRNet: Adaptive Graph Representation Learning and Reasoning for Face Parsingcited by 16
- 2023Toward High Quality Facial Representation Learningcited by 10
- 2023ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillationcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.