# "Are you okay, honey?": Recognizing Emotions among Couples Managing Diabetes in Daily Life using Multimodal Real-World Smartwatch Data

arXiv: https://arxiv.org/abs/2208.08909 · HTML: https://mlsanity.com/paper/2208.08909/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 65 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Emotion-Recognition Using Smart Watch Sensor Data: Mixed-Design Study](https://mlsanity.com/paper/1806.08518/) (2018)
- [BERT meets LIWC: Exploring State-of-the-Art Language Models for Predicting Communication Behavior in Couples' Conflict Interactions](https://mlsanity.com/paper/2106.01536/) (2021)
- [Automatic prediction of suicidal risk in military couples using multimodal interaction cues from couples conversations](https://mlsanity.com/paper/1911.11927/) (2019)
- [Predicting Behavior in Cancer-Afflicted Patient and Spouse Interactions using Speech and Language](https://mlsanity.com/paper/1908.00908/) (2019)
- [Modeling Interpersonal Influence of Verbal Behavior in Couples Therapy Dyadic Interactions](https://mlsanity.com/paper/1805.09436/) (2018)
- ["Making you happy makes me happy" -- Measuring Individual Mood with Smartwatches](https://mlsanity.com/paper/1711.06134/) (2017)
- ["You made me feel this way": Investigating Partners' Influence in Predicting Emotions in Couples' Conflict Interactions using Speech Data](https://mlsanity.com/paper/2106.01526/) (2021)
- [Emotion Recognition among Couples: A Survey](https://mlsanity.com/paper/2202.08430/) (2022)
- [Development, Deployment, and Evaluation of DyMand -- An Open-Source Smartwatch and Smartphone System for Capturing Couples' Dyadic Interactions in Chronic Disease Management in Daily Life](https://mlsanity.com/paper/2205.07671/) (2022)

## Abstract (arXiv)

Couples generally manage chronic diseases together and the management takes an emotional toll on both patients and their romantic partners. Consequently, recognizing the emotions of each partner in daily life could provide an insight into their emotional well-being in chronic disease management. Currently, the process of assessing each partner's emotions is manual, time-intensive, and costly. Despite the existence of works on emotion recognition among couples, none of these works have used data collected from couples' interactions in daily life. In this work, we collected 85 hours (1,021 5-minute samples) of real-world multimodal smartwatch sensor data (speech, heart rate, accelerometer, and gyroscope) and self-reported emotion data (n=612) from 26 partners (13 couples) managing diabetes mellitus type 2 in daily life. We extracted physiological, movement, acoustic, and linguistic features, and trained machine learning models (support vector machine and random forest) to recognize each partner's self-reported emotions (valence and arousal). Our results from the best models (balanced accuracies of 63.8% and 78.1% for arousal and valence respectively) are better than chance and our prior work that also used data from German-speaking, Swiss-based couples, albeit, in the lab. This work contributes toward building automated emotion recognition systems that would eventually enable partners to monitor their emotions in daily life and enable the delivery of interventions to improve their emotional well-being.