# 1DFormer: a Transformer Architecture Learning 1D Landmark Representations for Facial Landmark Tracking

arXiv: https://arxiv.org/abs/2311.00241 · HTML: https://mlsanity.com/paper/2311.00241/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Human Pose Regression with Residual Log-likelihood Estimation](https://mlsanity.com/paper/2107.11291/) (2021)
- [SimCC: a Simple Coordinate Classification Perspective for Human Pose Estimation](https://mlsanity.com/paper/2107.03332/) (2021)
- [Facial Landmark Detection: a Literature Survey](https://mlsanity.com/paper/1805.05563/) (2018)
- [Towards Accurate Facial Landmark Detection via Cascaded Transformers](https://mlsanity.com/paper/2208.10808/) (2022)
- [Face.evoLVe: A High-Performance Face Recognition Library](https://mlsanity.com/paper/2107.08621/) (2021)
- [Towards Highly Accurate and Stable Face Alignment for High-Resolution Videos](https://mlsanity.com/paper/1811.00342/) (2018)
- [RePFormer: Refinement Pyramid Transformer for Robust Facial Landmark Detection](https://mlsanity.com/paper/2207.03917/) (2022)
- [A Recurrent Encoder-Decoder Network for Sequential Face Alignment](https://mlsanity.com/paper/1608.05477/) (2016)
- [Gaussian Vector: An Efficient Solution for Facial Landmark Detection](https://mlsanity.com/paper/2010.01318/) (2020)
- [Robust and Precise Facial Landmark Detection by Self-Calibrated Pose Attention Network](https://mlsanity.com/paper/2112.12328/) (2021)
- [A Comprehensive Performance Evaluation of Deformable Face Tracking "In-the-Wild"](https://mlsanity.com/paper/1603.06015/) (2016)
- [Attentive One-Dimensional Heatmap Regression for Facial Landmark Detection and Tracking](https://mlsanity.com/paper/2004.02108/) (2020)
- [Reasoning Structural Relation for Occlusion-Robust Facial Landmark Localization](https://mlsanity.com/paper/2112.10087/) (2021)
- [A Hierarchical Probabilistic Model for Facial Feature Detection](https://mlsanity.com/paper/1709.05732/) (2017)

## Abstract (arXiv)

Recently, heatmap regression methods based on 1D landmark representations have shown prominent performance on locating facial landmarks. However, previous methods ignored to make deep explorations on the good potentials of 1D landmark representations for sequential and structural modeling of multiple landmarks to track facial landmarks. To address this limitation, we propose a Transformer architecture, namely 1DFormer, which learns informative 1D landmark representations by capturing the dynamic and the geometric patterns of landmarks via token communications in both temporal and spatial dimensions for facial landmark tracking. For temporal modeling, we propose a recurrent token mixing mechanism, an axis-landmark-positional embedding mechanism, as well as a confidence-enhanced multi-head attention mechanism to adaptively and robustly embed long-term landmark dynamics into their 1D representations; for structure modeling, we design intra-group and inter-group structure modeling mechanisms to encode the component-level as well as global-level facial structure patterns as a refinement for the 1D representations of landmarks through token communications in the spatial dimension via 1D convolutional layers. Experimental results on the 300VW and the TF databases show that 1DFormer successfully models the long-range sequential patterns as well as the inherent facial structures to learn informative 1D representations of landmark sequences, and achieves state-of-the-art performance on facial landmark tracking.