# Knowing When to Quit: Selective Cascaded Regression with Patch Attention for Real-Time Face Alignment

arXiv: https://arxiv.org/abs/2108.00377 · HTML: https://mlsanity.com/paper/2108.00377/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 1 Aug 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://mlsanity.com/paper/1904.07399/) (2019)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [LUVLi Face Alignment: Estimating Landmarks' Location, Uncertainty, and Visibility Likelihood](https://mlsanity.com/paper/2004.02980/) (2020)
- [DeCaFA: Deep Convolutional Cascade for Face Alignment In The Wild](https://mlsanity.com/paper/1904.02549/) (2019)
- [Deep Alignment Network: A convolutional neural network for robust face alignment](https://mlsanity.com/paper/1706.01789/) (2017)
- [Dense Face Alignment](https://mlsanity.com/paper/1709.01442/) (2017)
- [Semantic Alignment: Finding Semantically Consistent Ground-truth for Facial Landmark Detection](https://mlsanity.com/paper/1903.10661/) (2019)
- [Facial Landmarks Localization using Cascaded Neural Networks](https://mlsanity.com/paper/1805.01760/) (2018)
- [Facial Landmarks Detection by Self-Iterative Regression based Landmarks-Attention Network](https://mlsanity.com/paper/1803.06598/) (2018)

## Built on by (most cited first, 1 of 1)

- [Facial Landmark Points Detection Using Knowledge Distillation-Based Neural Networks](https://mlsanity.com/paper/2111.07047/) (2021)

## Abstract (arXiv)

Facial landmarks (FLM) estimation is a critical component in many face-related applications. In this work, we aim to optimize for both accuracy and speed and explore the trade-off between them. Our key observation is that not all faces are created equal. Frontal faces with neutral expressions converge faster than faces with extreme poses or expressions. To differentiate among samples, we train our model to predict the regression error after each iteration. If the current iteration is accurate enough, we stop iterating, saving redundant iterations while keeping the accuracy in check. We also observe that as neighboring patches overlap, we can infer all facial landmarks (FLMs) with only a small number of patches without a major accuracy sacrifice. Architecturally, we offer a multi-scale, patch-based, lightweight feature extractor with a fine-grained local patch attention module, which computes a patch weighting according to the information in the patch itself and enhances the expressive power of the patch features. We analyze the patch attention data to infer where the model is attending when regressing facial landmarks and compare it to face attention in humans. Our model runs in real-time on a mobile device GPU, with 95 Mega Multiply-Add (MMA) operations, outperforming all state-of-the-art methods under 1000 MMA, with a normalized mean error of 8.16 on the 300W challenging dataset.