# Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition

arXiv: https://arxiv.org/abs/2303.17611 · HTML: https://mlsanity.com/paper/2303.17611/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 51 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks](https://mlsanity.com/paper/1706.00527/) (2017)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [Uncovering the structure of clinical EEG signals with self-supervised learning](https://mlsanity.com/paper/2007.16104/) (2020)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [How Useful is Self-Supervised Pretraining for Visual Tasks?](https://mlsanity.com/paper/2003.14323/) (2020)
- [Subject-Aware Contrastive Learning for Biosignals](https://mlsanity.com/paper/2007.04871/) (2020)
- [A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances](https://mlsanity.com/paper/2203.06935/) (2022)
- [K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations](https://mlsanity.com/paper/2005.04120/) (2020)
- [Self-supervised Contrastive Learning for EEG-based Sleep Staging](https://mlsanity.com/paper/2109.07839/) (2021)
- [A dataset of continuous affect annotations and physiological signals for emotion analysis](https://mlsanity.com/paper/1812.02782/) (2018)
- [Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Data](https://mlsanity.com/paper/2008.10726/) (2020)
- [StressNAS: Affect State and Stress Detection Using Neural Architecture Search](https://mlsanity.com/paper/2108.12502/) (2021)
- [Sparse Fusion for Multimodal Transformers](https://mlsanity.com/paper/2111.11992/) (2021)

## Built on by (most cited first, 17 of 17)

- [A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications](https://mlsanity.com/paper/2408.02686/) (2024)
- [CLARE: Cognitive Load Assessment in REaltime with Multimodal Data](https://mlsanity.com/paper/2404.17098/) (2024)
- [Scaling Representation Learning from Ubiquitous ECG with State-Space Models](https://mlsanity.com/paper/2309.15292/) (2023)
- [Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention](https://mlsanity.com/paper/2407.18552/) (2024)
- [egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks](https://mlsanity.com/paper/2510.22129/) (2025)
- [Computational Analysis of Stress, Depression and Engagement in Mental Health: A Survey](https://mlsanity.com/paper/2403.08824/) (2024)
- [Robust Emotion Recognition via Bi-Level Self-Supervised Continual Learning](https://mlsanity.com/paper/2505.10575/) (2025)
- [Multi-scale Transformer-based Network for Emotion Recognition from Multi Physiological Signals](https://mlsanity.com/paper/2305.00769/) (2023)
- [Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection](https://mlsanity.com/paper/2606.24985/) (2026)
- [Wearable data from subjects playing Super Mario, sitting university exams, or performing physical exercise help detect acute mood episodes via self-supervised learning](https://mlsanity.com/paper/2311.04215/) (2023)
- [Learning Unified Representations from Heterogeneous Data for Robust Heart Rate Modeling](https://mlsanity.com/paper/2508.21785/) (2025)
- [Leveraging Vision Transformers for Enhanced Classification of Emotions using ECG Signals](https://mlsanity.com/paper/2510.05826/) (2025)
- [MIAR: Modality Interaction and Alignment Representation Fuison for Multimodal Emotion](https://mlsanity.com/paper/2601.02414/) (2026)
- [Encoding Emotion Through Self-Supervised Eye Movement Reconstruction](https://mlsanity.com/paper/2601.12534/) (2026)
- [Omni-modal decomposition autoencoders learn full-stack wearable disentangled representations](https://mlsanity.com/paper/2608.07385/) (2026)
- [Self-Supervised Graph Representation Learning for In-The-Wild Wearable and Smartphone based Emotion Recognition](https://mlsanity.com/paper/2608.22387/) (2026)
- [From Stress to Affect: Multimodal Deep Learning for Physiological Emotion Recognition Across Wearable Sensor Modalities](https://mlsanity.com/paper/2609.20991/) (2026)

## Abstract (arXiv)

Recently, wearable emotion recognition based on peripheral physiological signals has drawn massive attention due to its less invasive nature and its applicability in real-life scenarios. However, how to effectively fuse multimodal data remains a challenging problem. Moreover, traditional fully-supervised based approaches suffer from overfitting given limited labeled data. To address the above issues, we propose a novel self-supervised learning (SSL) framework for wearable emotion recognition, where efficient multimodal fusion is realized with temporal convolution-based modality-specific encoders and a transformer-based shared encoder, capturing both intra-modal and inter-modal correlations. Extensive unlabeled data is automatically assigned labels by five signal transforms, and the proposed SSL model is pre-trained with signal transformation recognition as a pretext task, allowing the extraction of generalized multimodal representations for emotion-related downstream tasks. For evaluation, the proposed SSL model was first pre-trained on a large-scale self-collected physiological dataset and the resulting encoder was subsequently frozen or fine-tuned on three public supervised emotion recognition datasets. Ultimately, our SSL-based method achieved state-of-the-art results in various emotion classification tasks. Meanwhile, the proposed model proved to be more accurate and robust compared to fully-supervised methods on low data regimes.