# K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

arXiv: https://arxiv.org/abs/2005.04120 · HTML: https://mlsanity.com/paper/2005.04120/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 30 of 30)

- [Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition](https://mlsanity.com/paper/2303.17611/) (2023)
- [A Tale of Single-channel Electroencephalogram: Devices, Datasets, Signal Processing, Applications, and Future Directions](https://mlsanity.com/paper/2407.14850/) (2024)
- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [Can Brain Signals Reveal Inner Alignment with Human Languages?](https://mlsanity.com/paper/2208.06348/) (2022)
- [Generalizable machine learning for stress monitoring from wearable devices: A systematic literature review](https://mlsanity.com/paper/2209.15137/) (2022)
- [A Comparison of Personalized and Generalized Approaches to Emotion Recognition Using Consumer Wearable Devices: Machine Learning Study](https://mlsanity.com/paper/2308.14245/) (2023)
- [AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work](https://mlsanity.com/paper/2205.06887/) (2022)
- [GameVibe: A Multimodal Affective Game Corpus](https://mlsanity.com/paper/2407.12787/) (2024)
- [Emotions in the Loop: A Survey of Affective Computing for Emotional Support](https://mlsanity.com/paper/2505.01542/) (2025)
- [egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks](https://mlsanity.com/paper/2510.22129/) (2025)
- [Understanding occupants' behaviour, engagement, emotion, and comfort indoors with heterogeneous sensors and wearables](https://mlsanity.com/paper/2105.06637/) (2021)
- [A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications](https://mlsanity.com/paper/2410.08224/) (2024)
- [A Valid Self-Report is Never Late, Nor is it Early: On Considering the "Right" Temporal Distance for Assessing Emotional Experience](https://mlsanity.com/paper/2302.02821/) (2023)
- [EmoSens: Emotion Recognition based on Sensor data analysis using LightGBM](https://mlsanity.com/paper/2207.14640/) (2022)
- [MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective Computing](https://mlsanity.com/paper/2409.07256/) (2024)
- [Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning](https://mlsanity.com/paper/2602.02784/) (2026)
- [AffectAI-Capture: A Reproducible Multimodal Protocol for Small-Group Meeting Research](https://mlsanity.com/paper/2605.19794/) (2026)
- [Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection](https://mlsanity.com/paper/2606.24985/) (2026)
- [Data Fusion in Neuromarketing: Multimodal Analysis of Biosignals, Lifecycle Stages, Current Advances, Datasets, Trends, and Challenges](https://mlsanity.com/paper/2209.00993/) (2022)
- [Decoding Emotional Valence from Wearables: Can Our Data Reveal Our True Feelings?](https://mlsanity.com/paper/2401.05408/) (2023)
- [CochCeps-Augment: A Novel Self-Supervised Contrastive Learning Using Cochlear Cepstrum-based Masking for Speech Emotion Recognition](https://mlsanity.com/paper/2402.06923/) (2024)
- [CuentosIE: can a chatbot about "tales with a message" help to teach emotional intelligence?](https://mlsanity.com/paper/2403.07193/) (2024)
- [CG-MER: A Card Game-based Multimodal dataset for Emotion Recognition](https://mlsanity.com/paper/2501.08182/) (2025)
- [AnnoSense: A Framework for Physiological Emotion Data Collection in Everyday Settings for AI](https://mlsanity.com/paper/2508.02680/) (2025)
- [MECO: A Multimodal Dataset for Emotion and Cognitive Understanding in Older Adults](https://mlsanity.com/paper/2604.03050/) (2026)
- [GroupAffect-4: A Multimodal Dataset of Four-Person Collaborative Interaction](https://mlsanity.com/paper/2605.19765/) (2026)
- [Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2605.21565/) (2026)
- [Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment](https://mlsanity.com/paper/2606.11269/) (2026)
- [Pseudo-Label Augmentation for Affect Sensing in Small Collaborative Groups](https://mlsanity.com/paper/2609.16077/) (2026)
- [From Momentary Emotion Inference to Sustained Emotion Support: Evaluating a Companion Agent in a Longitudinal Study](https://mlsanity.com/paper/2609.16344/) (2026)

## Abstract (arXiv)

Recognizing emotions during social interactions has many potential applications with the popularization of low-cost mobile sensors, but a challenge remains with the lack of naturalistic affective interaction data. Most existing emotion datasets do not support studying idiosyncratic emotions arising in the wild as they were collected in constrained environments. Therefore, studying emotions in the context of social interactions requires a novel dataset, and K-EmoCon is such a multimodal dataset with comprehensive annotations of continuous emotions during naturalistic conversations. The dataset contains multimodal measurements, including audiovisual recordings, EEG, and peripheral physiological signals, acquired with off-the-shelf devices from 16 sessions of approximately 10-minute long paired debates on a social issue. Distinct from previous datasets, it includes emotion annotations from all three available perspectives: self, debate partner, and external observers. Raters annotated emotional displays at intervals of every 5 seconds while viewing the debate footage, in terms of arousal-valence and 18 additional categorical emotions. The resulting K-EmoCon is the first publicly available emotion dataset accommodating the multiperspective assessment of emotions during social interactions.