# Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition

arXiv: https://arxiv.org/abs/2209.12172 · HTML: https://mlsanity.com/paper/2209.12172/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 63 references that are in the map.
- First posted on arXiv 25 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration](https://mlsanity.com/paper/1705.09634/) (2017)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [Understanding the Failure Modes of Out-of-Distribution Generalization](https://mlsanity.com/paper/2010.15775/) (2020)
- [Adversarially Adaptive Normalization for Single Domain Generalization](https://mlsanity.com/paper/2106.01899/) (2021)
- [Boosting Domain Adaptation by Discovering Latent Domains](https://mlsanity.com/paper/1805.01386/) (2018)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction](https://mlsanity.com/paper/1808.07773/) (2018)
- [MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition](https://mlsanity.com/paper/1911.09784/) (2019)
- [Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition](https://mlsanity.com/paper/2103.13372/) (2021)
- [Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport](https://mlsanity.com/paper/2111.06316/) (2021)
- [BReG-NeXt: Facial Affect Computing Using Adaptive Residual Networks With Bounded Gradient](https://mlsanity.com/paper/2004.08495/) (2020)
- [Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groups](https://mlsanity.com/paper/2110.09168/) (2021)
- [Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks](https://mlsanity.com/paper/1705.07884/) (2017)

## Built on by (most cited first, 8 of 8)

- [Norface: Improving Facial Expression Analysis by Identity Normalization](https://mlsanity.com/paper/2407.15617/) (2024)
- [A multimodal dynamical variational autoencoder for audiovisual speech representation learning](https://mlsanity.com/paper/2305.03582/) (2023)
- [OTCXR: Rethinking Self-supervised Alignment using Optimal Transport for Chest X-ray Analysis](https://mlsanity.com/paper/2404.11868/) (2024)
- [ReactDiff: Latent Diffusion for Facial Reaction Generation](https://mlsanity.com/paper/2505.14151/) (2025)
- [ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion](https://mlsanity.com/paper/2510.04706/) (2025)
- [Reverse Personalization](https://mlsanity.com/paper/2512.22984/) (2025)
- [The Circumplex Degeneracy Behind the Rare-Class Limit in Affect Recognition](https://mlsanity.com/paper/2606.15763/) (2026)
- [Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection](https://mlsanity.com/paper/2511.03666/) (2025)

## Abstract (arXiv)

Identity-invariant facial expression recognition (FER) has been one of the challenging computer vision tasks. Since conventional FER schemes do not explicitly address the inter-identity variation of facial expressions, their neural network models still operate depending on facial identity. This paper proposes to quantify the inter-identity variation by utilizing pairs of similar expressions explored through a specific matching process. We formulate the identity matching process as an Optimal Transport (OT) problem. Specifically, to find pairs of similar expressions from different identities, we define the inter-feature similarity as a transportation cost. Then, optimal identity matching to find the optimal flow with minimum transportation cost is performed by Sinkhorn-Knopp iteration. The proposed matching method is not only easy to plug in to other models, but also requires only acceptable computational overhead. Extensive simulations prove that the proposed FER method improves the PCC/CCC performance by up to 10% or more compared to the runner-up on wild datasets. The source code and software demo are available at https://github.com/kdhht2334/ELIMFER.