# Transformer-Based Self-Supervised Learning for Emotion Recognition

arXiv: https://arxiv.org/abs/2204.05103 · HTML: https://mlsanity.com/paper/2204.05103/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Self-supervised learning of a facial attribute embedding from video](https://mlsanity.com/paper/1808.06882/) (2018)
- [Self-supervised Learning for ECG-based Emotion Recognition](https://mlsanity.com/paper/1910.07497/) (2019)
- [A Transformer Architecture for Stress Detection from ECG](https://mlsanity.com/paper/2108.09737/) (2021)
- [Self-supervised Contrastive Learning of Multi-view Facial Expressions](https://mlsanity.com/paper/2108.06723/) (2021)
- [Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing](https://mlsanity.com/paper/1905.07039/) (2019)
- [On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognition](https://mlsanity.com/paper/2011.09212/) (2020)
- [Self-Supervised learning with cross-modal transformers for emotion recognition](https://mlsanity.com/paper/2011.10652/) (2020)
- [A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat](https://mlsanity.com/paper/1902.03043/) (2019)
- [Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Data](https://mlsanity.com/paper/2008.10726/) (2020)
- [Attending to Emotional Narratives](https://mlsanity.com/paper/1907.04197/) (2019)
- [Multi-modal embeddings using multi-task learning for emotion recognition](https://mlsanity.com/paper/2009.05019/) (2020)

## Built on by (most cited first, 16 of 16)

- [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023)
- [MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network](https://mlsanity.com/paper/2503.12623/) (2025)
- [Scaling Representation Learning from Ubiquitous ECG with State-Space Models](https://mlsanity.com/paper/2309.15292/) (2023)
- [Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals](https://mlsanity.com/paper/2212.13885/) (2022)
- [Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition](https://mlsanity.com/paper/2507.14867/) (2025)
- [Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learning](https://mlsanity.com/paper/2312.00502/) (2023)
- [MVP: Multimodal Emotion Recognition based on Video and Physiological Signals](https://mlsanity.com/paper/2501.03103/) (2025)
- [Robust Emotion Recognition via Bi-Level Self-Supervised Continual Learning](https://mlsanity.com/paper/2505.10575/) (2025)
- [Multi-scale Transformer-based Network for Emotion Recognition from Multi Physiological Signals](https://mlsanity.com/paper/2305.00769/) (2023)
- [Stress Classification from ECG Signals Using Vision Transformer](https://mlsanity.com/paper/2603.26721/) (2026)
- [AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses](https://mlsanity.com/paper/2405.13289/) (2024)
- [LLDif: Diffusion Models for Low-light Emotion Recognition](https://mlsanity.com/paper/2408.04235/) (2024)
- [Versatile and Risk-Sensitive Cardiac Diagnosis via Graph-Based ECG Signal Representation](https://mlsanity.com/paper/2511.07973/) (2025)
- [Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loads](https://mlsanity.com/paper/2511.22362/) (2025)
- [PPG-Based Affect Recognition with Long-Range Deep Models: A Measurement-Driven Comparison of CNN, Transformer, and Mamba Architectures](https://mlsanity.com/paper/2604.26078/) (2026)
- [Take it Personally: The Limits of General SSL Representations for Real-Life PPG Emotion Detection](https://mlsanity.com/paper/2608.14675/) (2026)

## Abstract (arXiv)

In order to exploit representations of time-series signals, such as physiological signals, it is essential that these representations capture relevant information from the whole signal. In this work, we propose to use a Transformer-based model to process electrocardiograms (ECG) for emotion recognition. Attention mechanisms of the Transformer can be used to build contextualized representations for a signal, giving more importance to relevant parts. These representations may then be processed with a fully-connected network to predict emotions. To overcome the relatively small size of datasets with emotional labels, we employ self-supervised learning. We gathered several ECG datasets with no labels of emotion to pre-train our model, which we then fine-tuned for emotion recognition on the AMIGOS dataset. We show that our approach reaches state-of-the-art performances for emotion recognition using ECG signals on AMIGOS. More generally, our experiments show that transformers and pre-training are promising strategies for emotion recognition with physiological signals.