# From the Lab to the Wild: Affect Modeling via Privileged Information

arXiv: https://arxiv.org/abs/2305.10919 · HTML: https://mlsanity.com/paper/2305.10919/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 81 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition](https://mlsanity.com/paper/2010.00734/) (2020)
- [Selective Sensor Fusion for Neural Visual-Inertial Odometry](https://mlsanity.com/paper/1903.01534/) (2019)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training](https://mlsanity.com/paper/1812.06145/) (2018)
- [The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games](https://mlsanity.com/paper/2101.10706/) (2021)
- [From Pixels to Affect: A Study on Games and Player Experience](https://mlsanity.com/paper/1907.02288/) (2019)
- [Detecting expressions with multimodal transformers](https://mlsanity.com/paper/2012.00063/) (2020)
- [Privileged Information for Modeling Affect In The Wild](https://mlsanity.com/paper/2107.10552/) (2021)
- [M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentation](https://mlsanity.com/paper/2205.02524/) (2022)
- [A Deep Learning Perspective on the Origin of Facial Expressions](https://mlsanity.com/paper/1705.01842/) (2017)
- [Dynamic Difficulty Awareness Training for Continuous Emotion Prediction](https://mlsanity.com/paper/1810.05507/) (2018)
- [Multimodal Emotion Recognition for One-Minute-Gradual Emotion Challenge](https://mlsanity.com/paper/1805.01060/) (2018)
- [Adversarial-based neural networks for affect estimations in the wild](https://mlsanity.com/paper/2002.00883/) (2020)

## Built on by (most cited first, 12 of 12)

- [Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport](https://mlsanity.com/paper/2401.15489/) (2024)
- [Across-Game Engagement Modelling via Few-Shot Learning](https://mlsanity.com/paper/2409.13002/) (2024)
- [Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition](https://mlsanity.com/paper/2408.09035/) (2024)
- [Privileged Contrastive Pretraining for Multimodal Affect Modelling](https://mlsanity.com/paper/2508.03729/) (2025)
- [Learning Using Privileged Information for Litter Detection](https://mlsanity.com/paper/2508.04124/) (2025)
- [Enhancing Object Detection with Privileged Information: A Model-Agnostic Teacher-Student Approach](https://mlsanity.com/paper/2601.02016/) (2026)
- [Learning using privileged information for segmenting tumors on digital mammograms](https://mlsanity.com/paper/2402.06379/) (2024)
- [PULSE: Privileged Knowledge Transfer from Rich to Deployable Sensors for Embodied Multi-Sensory Learning](https://mlsanity.com/paper/2510.24058/) (2025)
- [LaScA: Language-Conditioned Scalable Modelling of Affective Dynamics](https://mlsanity.com/paper/2604.07193/) (2026)
- [Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction](https://mlsanity.com/paper/2604.07198/) (2026)
- [Resource-Constrained Affect Modelling via Variance Regularisation Pruning](https://mlsanity.com/paper/2605.27479/) (2026)
- [BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition](https://mlsanity.com/paper/2608.06023/) (2026)

## Abstract (arXiv)

How can we reliably transfer affect models trained in controlled laboratory conditions (in-vitro) to uncontrolled real-world settings (in-vivo)? The information gap between in-vitro and in-vivo applications defines a core challenge of affective computing. This gap is caused by limitations related to affect sensing including intrusiveness, hardware malfunctions and availability of sensors. As a response to these limitations, we introduce the concept of privileged information for operating affect models in real-world scenarios (in the wild). Privileged information enables affect models to be trained across multiple modalities available in a lab, and ignore, without significant performance drops, those modalities that are not available when they operate in the wild. Our approach is tested in two multimodal affect databases one of which is designed for testing models of affect in the wild. By training our affect models using all modalities and then using solely raw footage frames for testing the models, we reach the performance of models that fuse all available modalities for both training and testing. The results are robust across both classification and regression affect modeling tasks which are dominant paradigms in affective computing. Our findings make a decisive step towards realizing affect interaction in the wild.