# Self-Supervised Approach for Facial Movement Based Optical Flow

arXiv: https://arxiv.org/abs/2105.01256 · HTML: https://mlsanity.com/paper/2105.01256/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 76 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 15 of 15)

- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)
- [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016)
- [Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation](https://mlsanity.com/paper/1808.01838/) (2018)
- [OFF-ApexNet on Micro-expression Recognition System](https://mlsanity.com/paper/1805.08699/) (2018)
- [Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition](https://mlsanity.com/paper/1902.03634/) (2019)
- [Micro-expression detection in long videos using optical flow and recurrent neural networks](https://mlsanity.com/paper/1903.10765/) (2019)
- [AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection](https://mlsanity.com/paper/1812.05788/) (2018)
- [Unsupervised Learning of Eye Gaze Representation from the Web](https://mlsanity.com/paper/1904.02459/) (2019)
- [Multi-View Dynamic Facial Action Unit Detection](https://mlsanity.com/paper/1704.07863/) (2017)
- [DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation](https://mlsanity.com/paper/2005.07298/) (2020)

## Built on by (most cited first, 3 of 3)

- [Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition](https://mlsanity.com/paper/2204.07636/) (2022)
- [FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model](https://mlsanity.com/paper/2409.05396/) (2024)
- [SRFlow: A Dataset and Regularization Model for High-Resolution Facial Optical Flow via Splatting Rasterization](https://mlsanity.com/paper/2601.06479/) (2026)

## Abstract (arXiv)

Computing optical flow is a fundamental problem in computer vision. However, deep learning-based optical flow techniques do not perform well for non-rigid movements such as those found in faces, primarily due to lack of the training data representing the fine facial motion. We hypothesize that learning optical flow on face motion data will improve the quality of predicted flow on faces. The aim of this work is threefold: (1) exploring self-supervised techniques to generate optical flow ground truth for face images; (2) computing baseline results on the effects of using face data to train Convolutional Neural Networks (CNN) for predicting optical flow; and (3) using the learned optical flow in micro-expression recognition to demonstrate its effectiveness. We generate optical flow ground truth using facial key-points in the BP4D-Spontaneous dataset. The generated optical flow is used to train the FlowNetS architecture to test its performance on the generated dataset. The performance of FlowNetS trained on face images surpassed that of other optical flow CNN architectures, demonstrating its usefulness. Our optical flow features are further compared with other methods using the STSTNet micro-expression classifier, and the results indicate that the optical flow obtained using this work has promising applications in facial expression analysis.