# Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels

arXiv: https://arxiv.org/abs/2101.03477 · HTML: https://mlsanity.com/paper/2101.03477/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 113 references that are in the map.
- First posted on arXiv 10 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions](https://mlsanity.com/paper/1801.02546/) (2018)
- [Deep learning for affective computing: text-based emotion recognition in decision support](https://mlsanity.com/paper/1803.06397/) (2018)
- [Superpower Glass: Delivering Unobtrusive Real-time Social Cues in Wearable Systems](https://mlsanity.com/paper/2002.06581/) (2020)
- [Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study](https://mlsanity.com/paper/2012.08678/) (2020)
- [Multi-Region Ensemble Convolutional Neural Network for Facial Expression Recognition](https://mlsanity.com/paper/1807.10575/) (2018)
- [Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognition](https://mlsanity.com/paper/2012.05429/) (2020)
- [Addressing Ambiguity of Emotion Labels Through Meta-Learning](https://mlsanity.com/paper/1911.02216/) (2019)

## Built on by (most cited first, 8 of 8)

- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Classification of Abnormal Hand Movement for Aiding in Autism Detection: Machine Learning Study](https://mlsanity.com/paper/2108.07917/) (2021)
- [Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy](https://mlsanity.com/paper/2510.10328/) (2025)
- [Addressing Racial Bias in Facial Emotion Recognition](https://mlsanity.com/paper/2308.04674/) (2023)
- [Challenges and Opportunities for Machine Learning Classification of Behavior and Mental State from Images](https://mlsanity.com/paper/2201.11197/) (2022)
- [Computer Vision Estimation of Emotion Reaction Intensity in the Wild](https://mlsanity.com/paper/2303.10741/) (2023)
- [Classifying Autism from Crowdsourced Semi-Structured Speech Recordings: A Machine Learning Approach](https://mlsanity.com/paper/2201.00927/) (2022)
- [An Exploration of Active Learning for Affective Digital Phenotyping](https://mlsanity.com/paper/2204.01915/) (2022)

## Abstract (arXiv)

Emotion classifiers traditionally predict discrete emotions. However, emotion expressions are often subjective, thus requiring a method to handle subjective labels. We explore the use of crowdsourcing to acquire reliable soft-target labels and evaluate an emotion detection classifier trained with these labels. We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard collection of images depicting pediatric facial expressions along with 100 human labels per image. To test the feasibility of crowdsourcing to generate these labels, we used Microworkers to acquire labels for 207 CAFE images. We evaluate both unfiltered workers as well as workers selected through a short crowd filtration process. We then train two versions of a classifiers on soft-target CAFE labels using the original 100 annotations provided with the dataset: (1) a classifier trained with traditional one-hot encoded labels, and (2) a classifier trained with vector labels representing the distribution of CAFE annotator responses. We compare the resulting softmax output distributions of the two classifiers with a 2-sample independent t-test of L1 distances between the classifier's output probability distribution and the distribution of human labels. While agreement with CAFE is weak for unfiltered crowd workers, the filtered crowd agree with the CAFE labels 100% of the time for many emotions. While the F1-score for a one-hot encoded classifier is much higher (94.33% vs. 78.68%) with respect to the ground truth CAFE labels, the output probability vector of the crowd-trained classifier more closely resembles the distribution of human labels (t=3.2827, p=0.0014). Reporting an emotion probability distribution that accounts for the subjectivity of human interpretation. Crowdsourcing, including a sufficient filtering mechanism, is a feasible solution for acquiring soft-target labels.