# Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning

arXiv: https://arxiv.org/abs/2111.15271 · HTML: https://mlsanity.com/paper/2111.15271/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [Context Based Emotion Recognition using EMOTIC Dataset](https://mlsanity.com/paper/2003.13401/) (2020)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates](https://mlsanity.com/paper/1706.08276/) (2017)
- [EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principle](https://mlsanity.com/paper/2003.06692/) (2020)
- [Facial expression and attributes recognition based on multi-task learning of lightweight neural networks](https://mlsanity.com/paper/2103.17107/) (2021)
- [SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition](https://mlsanity.com/paper/2004.11085/) (2020)
- [Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition](https://mlsanity.com/paper/2106.03487/) (2021)
- [One-shot Face Recognition by Promoting Underrepresented Classes](https://mlsanity.com/paper/1707.05574/) (2017)

## Built on by (most cited first, 1 of 1)

- [A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networks](https://mlsanity.com/paper/2501.07746/) (2025)

## Abstract (arXiv)

Human affect recognition is a well-established research area with numerous applications, e.g., in psychological care, but existing methods assume that all emotions-of-interest are given a priori as annotated training examples. However, the rising granularity and refinements of the human emotional spectrum through novel psychological theories and the increased consideration of emotions in context brings considerable pressure to data collection and labeling work. In this paper, we conceptualize one-shot recognition of emotions in context -- a new problem aimed at recognizing human affect states in finer particle level from a single support sample. To address this challenging task, we follow the deep metric learning paradigm and introduce a multi-modal emotion embedding approach which minimizes the distance of the same-emotion embeddings by leveraging complementary information of human appearance and the semantic scene context obtained through a semantic segmentation network. All streams of our context-aware model are optimized jointly using weighted triplet loss and weighted cross entropy loss. We conduct thorough experiments on both, categorical and numerical emotion recognition tasks of the Emotic dataset adapted to our one-shot recognition problem, revealing that categorizing human affect from a single example is a hard task. Still, all variants of our model clearly outperform the random baseline, while leveraging the semantic scene context consistently improves the learnt representations, setting state-of-the-art results in one-shot emotion recognition. To foster research of more universal representations of human affect states, we will make our benchmark and models publicly available to the community under https://github.com/KPeng9510/Affect-DML.