mlsanityData to 7 Oct 2026

Transformer-Based Self-Supervised Learning for Emotion Recognition

arXiv 2204.05103 ↗First posted 8 Apr 2022q-bio.NC cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

In order to exploit representations of time-series signals, such as physiological signals, it is essential that these representations capture relevant information from the whole signal. In this work, we propose to use a Transformer-based model to process electrocardiograms (ECG) for emotion recognition. Attention mechanisms of the Transformer can be used to build contextualized representations for a signal, giving more importance to relevant parts. These representations may then be processed with a fully-connected network to predict emotions. To overcome the relatively small size of datasets with emotional labels, we employ self-supervised learning. We gathered several ECG datasets with no labels of emotion to pre-train our model, which we then fine-tuned for emotion recognition on the AMIGOS dataset. We show that our approach reaches state-of-the-art performances for emotion recognition using ECG signals on AMIGOS. More generally, our experiments show that transformers and pre-training are promising strategies for emotion recognition with physiological signals.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 26 of its 61 references that are in the map. First posted on arXiv 8 Apr 2022.

Builds on

26 of 61 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  10. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  11. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  12. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
14 more
  1. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  2. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  3. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  4. 2018Self-supervised learning of a facial attribute embedding from videocited by 44
  5. 2019Self-supervised Learning for ECG-based Emotion Recognitioncited by 32
  6. 2021A Transformer Architecture for Stress Detection from ECGcited by 27
  7. 2021Self-supervised Contrastive Learning of Multi-view Facial Expressionscited by 23
  8. 2019Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computingcited by 22
  9. 2020On the use of Self-supervised Pre-trained Acoustic and Linguistic Features for Continuous Speech Emotion Recognitioncited by 19
  10. 2020Self-Supervised learning with cross-modal transformers for emotion recognitioncited by 12
  11. 2019A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeatcited by 11
  12. 2020Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Datacited by 10
  13. 2019Attending to Emotional Narrativescited by 5
  14. 2020Multi-modal embeddings using multi-task learning for emotion recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 2; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Modelscited by 16
  2. 2025MAVEN: Multi-modal Attention for Valence-Arousal Emotion Networkcited by 8
  3. 2023Scaling Representation Learning from Ubiquitous ECG with State-Space Modelscited by 7
  4. 2022Emotion Recognition with Pre-Trained Transformers Using Multimodal Signalscited by 6
  5. 2025Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognitioncited by 3
  6. 2023Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learningcited by 2
  7. 2025MVP: Multimodal Emotion Recognition based on Video and Physiological Signalscited by 2
  8. 2025Robust Emotion Recognition via Bi-Level Self-Supervised Continual Learningcited by 2
  9. 2023Multi-scale Transformer-based Network for Emotion Recognition from Multi Physiological Signalscited by 1
  10. 2026Stress Classification from ECG Signals Using Vision Transformercited by 1
  11. 2024AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glassescited by 0
  12. 2024LLDif: Diffusion Models for Low-light Emotion Recognitioncited by 0
4 more
  1. 2025Versatile and Risk-Sensitive Cardiac Diagnosis via Graph-Based ECG Signal Representationcited by 0
  2. 2025Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loadscited by 0
  3. 2026PPG-Based Affect Recognition with Long-Range Deep Models: A Measurement-Driven Comparison of CNN, Transformer, and Mamba Architecturescited by 0
  4. 2026Take it Personally: The Limits of General SSL Representations for Real-Life PPG Emotion Detectioncited by 0