# From Facial Expression Recognition to Interpersonal Relation Prediction

arXiv: https://arxiv.org/abs/1609.06426 · HTML: https://mlsanity.com/paper/1609.06426/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2016.

## Built on by (most cited first, 30 of 53)

- [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition](https://mlsanity.com/paper/2212.05463/) (2022)
- [Py-Feat: Python Facial Expression Analysis Toolbox](https://mlsanity.com/paper/2104.03509/) (2021)
- [Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition](https://mlsanity.com/paper/2108.04228/) (2021)
- [Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions](https://mlsanity.com/paper/2109.09487/) (2021)
- [Graph-based Facial Affect Analysis: A Review](https://mlsanity.com/paper/2103.15599/) (2021)
- [ReSup: Reliable Label Noise Suppression for Facial Expression Recognition](https://mlsanity.com/paper/2305.17895/) (2023)
- [Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition](https://mlsanity.com/paper/2303.15889/) (2023)
- [Multitask Multi-database Emotion Recognition](https://mlsanity.com/paper/2107.04127/) (2021)
- [Towards Online Multi-Modal Social Interaction Understanding](https://mlsanity.com/paper/2503.19851/) (2025)
- [Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes](https://mlsanity.com/paper/2307.05862/) (2023)
- [ResEmoteNet: Bridging Accuracy and Loss Reduction in Facial Emotion Recognition](https://mlsanity.com/paper/2409.10545/) (2024)
- [AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels](https://mlsanity.com/paper/2410.22506/) (2024)
- [A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directions](https://mlsanity.com/paper/2305.07665/) (2023)
- [Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models](https://mlsanity.com/paper/2502.11049/) (2025)
- [Training and Profiling a Pediatric Emotion Recognition Classifier on Mobile Devices](https://mlsanity.com/paper/2108.11754/) (2021)
- [Learning from Synthetic Data: Facial Expression Classification based on Ensemble of Multi-task Networks](https://mlsanity.com/paper/2207.10025/) (2022)
- [Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition](https://mlsanity.com/paper/2210.05246/) (2022)
- [DSAP: Analyzing Bias Through Demographic Comparison of Datasets](https://mlsanity.com/paper/2312.14626/) (2023)
- [Less can be more: representational vs. stereotypical gender bias in facial expression recognition](https://mlsanity.com/paper/2406.17405/) (2024)
- [Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey](https://mlsanity.com/paper/2412.17616/) (2024)
- [FaceTopoNet: Facial Expression Recognition using Face Topology Learning](https://mlsanity.com/paper/2209.06322/) (2022)
- [Exploring Large-scale Unlabeled Faces to Enhance Facial Expression Recognition](https://mlsanity.com/paper/2303.08617/) (2023)
- [Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categories](https://mlsanity.com/paper/2308.10562/) (2023)
- [Adaptive Global-Local Representation Learning and Selection for Cross-Domain Facial Expression Recognition](https://mlsanity.com/paper/2401.11085/) (2024)
- [SUN Team's Contribution to ABAW 2024 Competition: Audio-visual Valence-Arousal Estimation and Expression Recognition](https://mlsanity.com/paper/2403.12609/) (2024)
- [VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligence](https://mlsanity.com/paper/2504.02227/) (2025)
- [Feature Pyramid Network for Multi-task Affective Analysis](https://mlsanity.com/paper/2107.03670/) (2021)
- [Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation](https://mlsanity.com/paper/2409.08598/) (2024)

## Abstract (arXiv)

Interpersonal relation defines the association, e.g., warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterized and quantified from face images in the wild. We address this challenging problem by first studying a deep network architecture for robust recognition of facial expressions. Unlike existing models that typically learn from facial expression labels alone, we devise an effective multitask network that is capable of learning from rich auxiliary attributes such as gender, age, and head pose, beyond just facial expression data. While conventional supervised training requires datasets with complete labels (e.g., all samples must be labeled with gender, age, and expression), we show that this requirement can be relaxed via a novel attribute propagation method. The approach further allows us to leverage the inherent correspondences between heterogeneous attribute sources despite the disparate distributions of different datasets. With the network we demonstrate state-of-the-art results on existing facial expression recognition benchmarks. To predict inter-personal relation, we use the expression recognition network as branches for a Siamese model. Extensive experiments show that our model is capable of mining mutual context of faces for accurate fine-grained interpersonal prediction.