# PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation

arXiv: https://arxiv.org/abs/2108.10710 · HTML: https://mlsanity.com/paper/2108.10710/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 24 Aug 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [MixConv: Mixed Depthwise Convolutional Kernels](https://mlsanity.com/paper/1907.09595/) (2019)
- [One-Shot Neural Architecture Search via Self-Evaluated Template Network](https://mlsanity.com/paper/1910.05733/) (2019)
- [MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation](https://mlsanity.com/paper/2003.12238/) (2020)
- [MixFaceNets: Extremely Efficient Face Recognition Networks](https://mlsanity.com/paper/2107.13046/) (2021)
- [VarGFaceNet: An Efficient Variable Group Convolutional Neural Network for Lightweight Face Recognition](https://mlsanity.com/paper/1910.04985/) (2019)
- [ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search](https://mlsanity.com/paper/2105.10154/) (2021)

## Built on by (most cited first, 30 of 36)

- [IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models](https://mlsanity.com/paper/2308.04995/) (2023)
- [SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data](https://mlsanity.com/paper/2206.10520/) (2022)
- [MorDIFF: Recognition Vulnerability and Attack Detectability of Face Morphing Attacks Created by Diffusion Autoencoders](https://mlsanity.com/paper/2302.01843/) (2023)
- [EdgeFace: Efficient Face Recognition Model for Edge Devices](https://mlsanity.com/paper/2307.01838/) (2023)
- [Identity-driven Three-Player Generative Adversarial Network for Synthetic-based Face Recognition](https://mlsanity.com/paper/2305.00358/) (2023)
- [QuantFace: Towards Lightweight Face Recognition by Synthetic Data Low-bit Quantization](https://mlsanity.com/paper/2206.10526/) (2022)
- [Efficient Explainable Face Verification based on Similarity Score Argument Backpropagation](https://mlsanity.com/paper/2304.13409/) (2023)
- [Data Synthesis with Diverse Styles for Face Recognition via 3DMM-Guided Diffusion](https://mlsanity.com/paper/2504.00430/) (2025)
- [Mask-invariant Face Recognition through Template-level Knowledge Distillation](https://mlsanity.com/paper/2112.05646/) (2021)
- [SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data](https://mlsanity.com/paper/2308.14852/) (2023)
- [Face Morphing Attacks and Face Image Quality: The Effect of Morphing and the Unsupervised Attack Detection by Quality](https://mlsanity.com/paper/2208.05864/) (2022)
- [The Effect of Wearing a Face Mask on Face Image Quality](https://mlsanity.com/paper/2110.11283/) (2021)
- [OCFR 2022: Competition on Occluded Face Recognition From Synthetically Generated Structure-Aware Occlusions](https://mlsanity.com/paper/2208.02760/) (2022)
- [Towards Explaining Demographic Bias through the Eyes of Face Recognition Models](https://mlsanity.com/paper/2208.13400/) (2022)
- [EFaR 2023: Efficient Face Recognition Competition](https://mlsanity.com/paper/2308.04168/) (2023)
- [Model Compression Techniques in Biometrics Applications: A Survey](https://mlsanity.com/paper/2401.10139/) (2024)
- [xEdgeFace: Efficient Cross-Spectral Face Recognition for Edge Devices](https://mlsanity.com/paper/2504.19646/) (2025)
- [If It's Not Enough, Make It So: Reducing Authentic Data Demand in Face Recognition through Synthetic Faces](https://mlsanity.com/paper/2404.03537/) (2024)
- [A Survey on Drowsiness Detection -- Modern Applications and Methods](https://mlsanity.com/paper/2408.12990/) (2024)
- [Young Labeled Faces in the Wild (YLFW): A Dataset for Children Faces Recognition](https://mlsanity.com/paper/2301.05776/) (2023)
- [Deep Network Pruning: A Comparative Study on CNNs in Face Recognition](https://mlsanity.com/paper/2405.18302/) (2024)
- [The Impact of Balancing Real and Synthetic Data on Accuracy and Fairness in Face Recognition](https://mlsanity.com/paper/2409.02867/) (2024)
- [UIFace: Unleashing Inherent Model Capabilities to Enhance Intra-Class Diversity in Synthetic Face Recognition](https://mlsanity.com/paper/2502.19803/) (2025)
- [FaceLiVTv2: An Improved Hybrid Architecture for Efficient Mobile Face Recognition](https://mlsanity.com/paper/2604.09127/) (2026)
- [GMFIM: A Generative Mask-guided Facial Image Manipulation Model for Privacy Preservation](https://mlsanity.com/paper/2201.03353/) (2022)
- [SqueezerFaceNet: Reducing a Small Face Recognition CNN Even More Via Filter Pruning](https://mlsanity.com/paper/2307.10697/) (2023)
- [Worst-Case Morphs using Wasserstein ALI and Improved MIPGAN](https://mlsanity.com/paper/2310.08371/) (2023)
- [MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition](https://mlsanity.com/paper/2408.16563/) (2024)
- [Efficient Verification-Based Face Identification](https://mlsanity.com/paper/2312.13240/) (2023)
- [How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition](https://mlsanity.com/paper/2408.17399/) (2024)

## Abstract (arXiv)

Deep neural networks have rapidly become the mainstream method for face recognition (FR). However, this limits the deployment of such models that contain an extremely large number of parameters to embedded and low-end devices. In this work, we present an extremely lightweight and accurate FR solution, namely PocketNet. We utilize neural architecture search to develop a new family of lightweight face-specific architectures. We additionally propose a novel training paradigm based on knowledge distillation (KD), the multi-step KD, where the knowledge is distilled from the teacher model to the student model at different stages of the training maturity. We conduct a detailed ablation study proving both, the sanity of using NAS for the specific task of FR rather than general object classification, and the benefits of our proposed multi-step KD. We present an extensive experimental evaluation and comparisons with the state-of-the-art (SOTA) compact FR models on nine different benchmarks including large-scale evaluation benchmarks such as IJB-B, IJB-C, and MegaFace. PocketNets have consistently advanced the SOTA FR performance on nine mainstream benchmarks when considering the same level of model compactness. With 0.92M parameters, our smallest network PocketNetS-128 achieved very competitive results to recent SOTA compacted models that contain up to 4M parameters.