# Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data

arXiv: https://arxiv.org/abs/2106.13213 · HTML: https://mlsanity.com/paper/2106.13213/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 59 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](https://mlsanity.com/paper/1907.09173/) (2019)
- [Threats to Federated Learning: A Survey](https://mlsanity.com/paper/2003.02133/) (2020)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Multimodal Language Analysis with Recurrent Multistage Fusion](https://mlsanity.com/paper/1808.03920/) (2018)
- [Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis](https://mlsanity.com/paper/1609.05244/) (2016)
- [Bias Amplification in Artificial Intelligence Systems](https://mlsanity.com/paper/1809.07842/) (2018)
- [DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detection](https://mlsanity.com/paper/1803.08986/) (2018)
- [dpMood: Exploiting Local and Periodic Typing Dynamics for Personalized Mood Prediction](https://mlsanity.com/paper/1808.09852/) (2018)

## Built on by (most cited first, 9 of 9)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources](https://mlsanity.com/paper/2305.06217/) (2023)
- [MultiMed: Massively Multimodal and Multitask Medical Understanding](https://mlsanity.com/paper/2408.12682/) (2024)
- [Neural Mixed Effects for Nonlinear Personalized Predictions](https://mlsanity.com/paper/2306.08149/) (2023)
- [MultiIoT: Benchmarking Machine Learning for the Internet of Things](https://mlsanity.com/paper/2311.06217/) (2023)
- [Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms](https://mlsanity.com/paper/2505.21792/) (2025)
- [A Vision for Multisensory Intelligence: Sensing, Science, and Synergy](https://mlsanity.com/paper/2601.04563/) (2026)

## Abstract (arXiv)

Mental health conditions remain underdiagnosed even in countries with common access to advanced medical care. The ability to accurately and efficiently predict mood from easily collectible data has several important implications for the early detection, intervention, and treatment of mental health disorders. One promising data source to help monitor human behavior is daily smartphone usage. However, care must be taken to summarize behaviors without identifying the user through personal (e.g., personally identifiable information) or protected (e.g., race, gender) attributes. In this paper, we study behavioral markers of daily mood using a recent dataset of mobile behaviors from adolescent populations at high risk of suicidal behaviors. Using computational models, we find that language and multimodal representations of mobile typed text (spanning typed characters, words, keystroke timings, and app usage) are predictive of daily mood. However, we find that models trained to predict mood often also capture private user identities in their intermediate representations. To tackle this problem, we evaluate approaches that obfuscate user identity while remaining predictive. By combining multimodal representations with privacy-preserving learning, we are able to push forward the performance-privacy frontier.