# Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learning

arXiv: https://arxiv.org/abs/2308.02173 · HTML: https://mlsanity.com/paper/2308.02173/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 64 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition](https://mlsanity.com/paper/2109.09559/) (2021)
- [Few-Shot Learning with Metric-Agnostic Conditional Embeddings](https://mlsanity.com/paper/1802.04376/) (2018)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [Spatiotemporal Contrastive Learning of Facial Expressions in Videos](https://mlsanity.com/paper/2108.03064/) (2021)
- [Speech Emotion Recognition via Contrastive Loss under Siamese Networks](https://mlsanity.com/paper/1910.11174/) (2019)
- [Demystification of Few-shot and One-shot Learning](https://mlsanity.com/paper/2104.12174/) (2021)
- [Multi-Task Learning Framework for Emotion Recognition in-the-wild](https://mlsanity.com/paper/2207.09373/) (2022)
- [Metric-Based Few-Shot Learning for Video Action Recognition](https://mlsanity.com/paper/1909.09602/) (2019)
- [Revisiting Few-Shot Learning for Facial Expression Recognition](https://mlsanity.com/paper/1912.02751/) (2019)
- [Few-shot Learning in Emotion Recognition of Spontaneous Speech Using a Siamese Neural Network with Adaptive Sample Pair Formation](https://mlsanity.com/paper/2109.02915/) (2021)
- [When Facial Expression Recognition Meets Few-Shot Learning: A Joint and Alternate Learning Framework](https://mlsanity.com/paper/2201.06781/) (2022)
- [Affect Recognition in Ads with Application to Computational Advertising](https://mlsanity.com/paper/1709.01683/) (2017)
- [Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild](https://mlsanity.com/paper/1902.04042/) (2019)
- [Gender and Emotion Recognition with Implicit User Signals](https://mlsanity.com/paper/1708.08735/) (2017)
- [To Improve Is to Change: Towards Improving Mood Prediction by Learning Changes in Emotion](https://mlsanity.com/paper/2210.00719/) (2022)
- [A Weakly Supervised Approach to Emotion-change Prediction and Improved Mood Inference](https://mlsanity.com/paper/2306.06979/) (2023)

## Built on by (most cited first, 2 of 2)

- [Rethinking Affect Analysis: A Protocol for Ensuring Fairness and Consistency](https://mlsanity.com/paper/2408.02164/) (2024)
- [Neuromorphic Valence and Arousal Estimation](https://mlsanity.com/paper/2401.16058/) (2024)

## Abstract (arXiv)

Whilst deep learning techniques have achieved excellent emotion prediction, they still require large amounts of labelled training data, which are (a) onerous and tedious to compile, and (b) prone to errors and biases. We propose Multi-Task Contrastive Learning for Affect Representation (MT-CLAR) for few-shot affect inference. MT-CLAR combines multi-task learning with a Siamese network trained via contrastive learning to infer from a pair of expressive facial images (a) the (dis)similarity between the facial expressions, and (b) the difference in valence and arousal levels of the two faces. We further extend the image-based MT-CLAR framework for automated video labelling where, given one or a few labelled video frames (termed textitsupport-set), MT-CLAR labels the remainder of the video for valence and arousal. Experiments are performed on the AFEW-VA dataset with multiple support-set configurations; moreover, supervised learning on representations learnt via MT-CLAR are used for valence, arousal and categorical emotion prediction on the AffectNet and AFEW-VA datasets. The results show that valence and arousal predictions via MT-CLAR are very comparable to the state-of-the-art (SOTA), and we significantly outperform SOTA with a support-set ≈6% the size of the video dataset.