Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data
Not on the map. Why?
Abstract (arXiv)
Mental health conditions remain underdiagnosed even in countries with common access to advanced medical care. The ability to accurately and efficiently predict mood from easily collectible data has several important implications for the early detection, intervention, and treatment of mental health disorders. One promising data source to help monitor human behavior is daily smartphone usage. However, care must be taken to summarize behaviors without identifying the user through personal (e.g., personally identifiable information) or protected (e.g., race, gender) attributes. In this paper, we study behavioral markers of daily mood using a recent dataset of mobile behaviors from adolescent populations at high risk of suicidal behaviors. Using computational models, we find that language and multimodal representations of mobile typed text (spanning typed characters, words, keystroke timings, and app usage) are predictive of daily mood. However, we find that models trained to predict mood often also capture private user identities in their intermediate representations. To tackle this problem, we evaluate approaches that obfuscate user identity while remaining predictive. By combining multimodal representations with privacy-preserving learning, we are able to push forward the performance-privacy frontier.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 21 of its 59 references that are in the map. First posted on arXiv 24 Jun 2021.
Builds on
21 of 59 references.
References · 21
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2016Deep Learning with Differential Privacycited by 2,682
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2019Deep Leakage from Gradientscited by 1,004
- 2017Federated Multi-Task Learningcited by 669
- 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
9 more
- 2019FedHealth: A Federated Transfer Learning Framework for Wearable Healthcarecited by 182
- 2020Threats to Federated Learning: A Surveycited by 171
- 2020Towards Debiasing Sentence Representationscited by 170
- 2019Inherent Tradeoffs in Learning Fair Representationscited by 144
- 2018Multimodal Language Analysis with Recurrent Multistage Fusioncited by 43
- 2016Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysiscited by 29
- 2018Bias Amplification in Artificial Intelligence Systemscited by 15
- 2018DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detectioncited by 10
- 2018dpMood: Exploiting Local and Periodic Typing Dynamics for Personalized Mood Predictioncited by 1
Built on by
Citing papers per year: 2022: 2; 2023: 4; 2024: 1; 2025: 1; 2026: 1.
9 papers.
- 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
- 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
- 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
- 2023Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sourcescited by 10
- 2024MultiMed: Massively Multimodal and Multitask Medical Understandingcited by 8
- 2023Neural Mixed Effects for Nonlinear Personalized Predictionscited by 5
- 2023MultiIoT: Benchmarking Machine Learning for the Internet of Thingscited by 1
- 2025Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigmscited by 0
- 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
- 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergy8 Jan 2026
- 2025Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms27 May 2025
- 2024MultiMed: Massively Multimodal and Multitask Medical Understanding22 Aug 2024
- 2023MultiIoT: Benchmarking Machine Learning for the Internet of Things10 Nov 2023
- 2023Neural Mixed Effects for Nonlinear Personalized Predictions13 Jun 2023
- 2023Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources10 May 2023
- 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework23 Feb 2023
- 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions7 Sep 2022
- 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning2 Mar 2022