# Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition

arXiv: https://arxiv.org/abs/2303.04896 · HTML: https://mlsanity.com/paper/2303.04896/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Gradient Starvation: A Learning Proclivity in Neural Networks](https://mlsanity.com/paper/2011.09468/) (2020)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations](https://mlsanity.com/paper/1707.00075/) (2017)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Learning explanations that are hard to vary](https://mlsanity.com/paper/2009.00329/) (2020)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)
- [50 Years of Test (Un)fairness: Lessons for Machine Learning](https://mlsanity.com/paper/1811.10104/) (2018)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://mlsanity.com/paper/1909.10681/) (2019)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [Shape-Texture Debiased Neural Network Training](https://mlsanity.com/paper/2010.05981/) (2020)
- [BiaSwap: Removing dataset bias with bias-tailored swapping augmentation](https://mlsanity.com/paper/2108.10008/) (2021)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification](https://mlsanity.com/paper/2109.05427/) (2021)
- [Characterizing Intersectional Group Fairness with Worst-Case Comparisons](https://mlsanity.com/paper/2101.01673/) (2021)
- [Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias](https://mlsanity.com/paper/1808.02212/) (2018)
- [Understanding and Improving Fairness-Accuracy Trade-offs in Multi-Task Learning](https://mlsanity.com/paper/2106.02705/) (2021)
- [Py-Feat: Python Facial Expression Analysis Toolbox](https://mlsanity.com/paper/2104.03509/) (2021)
- [Facial Motion Prior Networks for Facial Expression Recognition](https://mlsanity.com/paper/1902.08788/) (2019)
- [Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?](https://mlsanity.com/paper/1510.02969/) (2015)
- [From Standard Summarization to New Tasks and Beyond: Summarization with Manifold Information](https://mlsanity.com/paper/2005.04684/) (2020)
- [Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognition](https://mlsanity.com/paper/2108.00194/) (2021)

## Built on by (most cited first, 4 of 4)

- [Learning Contrastive Feature Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2402.06165/) (2024)
- [One-Frame Calibration with Siamese Network in Facial Action Unit Recognition](https://mlsanity.com/paper/2409.00240/) (2024)
- [Prior-based Objective Inference Mining Potential Uncertainty for Facial Expression Recognition](https://mlsanity.com/paper/2411.13024/) (2024)
- [A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models](https://mlsanity.com/paper/2505.20512/) (2025)

## Abstract (arXiv)

Machine learning models automatically learn discriminative features from the data, and are therefore susceptible to learn strongly-correlated biases, such as using protected attributes like gender and race. Most existing bias mitigation approaches aim to explicitly reduce the model's focus on these protected features. In this work, we propose to mitigate bias by explicitly guiding the model's focus towards task-relevant features using domain knowledge, and we hypothesize that this can indirectly reduce the dependence of the model on spurious correlations it learns from the data. We explore bias mitigation in facial expression recognition systems using facial Action Units (AUs) as the task-relevant feature. To this end, we introduce Feature-based Positive Matching Contrastive Loss which learns the distances between the positives of a sample based on the similarity between their corresponding AU embeddings. We compare our approach with representative baselines and show that incorporating task-relevant features via our method can improve model fairness at minimal cost to classification performance.