# The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games

arXiv: https://arxiv.org/abs/2101.10706 · HTML: https://mlsanity.com/paper/2101.10706/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2021.

## Built on by (most cited first, 15 of 15)

- [From the Lab to the Wild: Affect Modeling via Privileged Information](https://mlsanity.com/paper/2305.10919/) (2023)
- [Contrastive Learning of Generalized Game Representations](https://mlsanity.com/paper/2106.10060/) (2021)
- [Affective Game Computing: A Survey](https://mlsanity.com/paper/2309.14104/) (2023)
- [AffectGAN: Affect-Based Generative Art Driven by Semantics](https://mlsanity.com/paper/2109.14845/) (2021)
- [Supervised Contrastive Learning for Affect Modelling](https://mlsanity.com/paper/2208.12238/) (2022)
- [Go-Blend behavior and affect](https://mlsanity.com/paper/2109.13388/) (2021)
- [Learning Task-Independent Game State Representations from Unlabeled Images](https://mlsanity.com/paper/2206.06490/) (2022)
- [Play with Emotion: Affect-Driven Reinforcement Learning](https://mlsanity.com/paper/2208.12622/) (2022)
- [The Invariant Ground Truth of Affect](https://mlsanity.com/paper/2210.07630/) (2022)
- [RankNEAT: Outperforming Stochastic Gradient Search in Preference Learning Tasks](https://mlsanity.com/paper/2204.06901/) (2022)
- [AffRankNet+: Ranking Affect Using Privileged Information](https://mlsanity.com/paper/2108.05598/) (2021)
- [Privileged Contrastive Pretraining for Multimodal Affect Modelling](https://mlsanity.com/paper/2508.03729/) (2025)
- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [Closing the Affective Loop via Experience-Driven Reinforcement Learning Designers](https://mlsanity.com/paper/2408.06346/) (2024)
- [PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation](https://mlsanity.com/paper/2601.13904/) (2026)

## Abstract (arXiv)

What if emotion could be captured in a general and subject-agnostic fashion? Is it possible, for instance, to design general-purpose representations that detect affect solely from the pixels and audio of a human-computer interaction video? In this paper we address the above questions by evaluating the capacity of deep learned representations to predict affect by relying only on audiovisual information of videos. We assume that the pixels and audio of an interactive session embed the necessary information required to detect affect. We test our hypothesis in the domain of digital games and evaluate the degree to which deep classifiers and deep preference learning algorithms can learn to predict the arousal of players based only on the video footage of their gameplay. Our results from four dissimilar games suggest that general-purpose representations can be built across games as the arousal models obtain average accuracies as high as 85% using the challenging leave-one-video-out cross-validation scheme. The dissimilar audiovisual characteristics of the tested games showcase the strengths and limitations of the proposed method.