# From 'What' to 'How' and 'Why': Sharing LLM-Generated Retrospective Summaries of Older Adults' Passive Tracking Data with Remote Family Members

arXiv: https://arxiv.org/abs/2606.03876 · HTML: https://mlsanity.com/paper/2606.03876/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 96 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 15 of 15)

- [Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data](https://mlsanity.com/paper/2401.06866/) (2024)
- [LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces](https://mlsanity.com/paper/2403.19857/) (2024)
- [Towards a Personal Health Large Language Model](https://mlsanity.com/paper/2406.06474/) (2024)
- [DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledge](https://mlsanity.com/paper/2405.12541/) (2024)
- [MindScape Study: Integrating LLM and Behavioral Sensing for Personalized AI-Driven Journaling Experiences](https://mlsanity.com/paper/2409.09570/) (2024)
- [PhysioLLM: Supporting Personalized Health Insights with Wearables and Large Language Models](https://mlsanity.com/paper/2406.19283/) (2024)
- [Contextual AI Journaling: Integrating LLM and Time Series Behavioral Sensing Technology to Promote Self-Reflection and Well-being using the MindScape App](https://mlsanity.com/paper/2404.00487/) (2024)
- [Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM](https://mlsanity.com/paper/2410.14879/) (2024)
- [DailyLLM: Context-Aware Activity Log Generation Using Multi-Modal Sensors and LLMs](https://mlsanity.com/paper/2507.13737/) (2025)
- [AI-Enabled Conversational Journaling for Advancing Parkinson's Disease Symptom Tracking](https://mlsanity.com/paper/2503.03532/) (2025)
- [GLOSS: Group of LLMs for Open-Ended Sensemaking of Passive Sensing Data for Health and Wellbeing](https://mlsanity.com/paper/2507.05461/) (2025)
- [Effect of Selection Format on LLM Performance](https://mlsanity.com/paper/2503.06926/) (2025)
- [ConSensus: Multi-Agent Collaboration for Multimodal Sensing](https://mlsanity.com/paper/2601.06453/) (2026)
- [Categorizing Sources of Information for Explanations in Conversational AI Systems for Older Adults Aging in Place](https://mlsanity.com/paper/2406.05111/) (2024)
- [Bias Beneath the Tone: Empirical Characterisation of Tone Bias in LLM-Driven UX Systems](https://mlsanity.com/paper/2512.19950/) (2025)

## Abstract (arXiv)

With the growing prevalence of modern ubiquitous computing technologies, multi-modal tracking systems hold promise for providing timely awareness and reassurance to stakeholders such as remote family members (RFMs) of older adults, who play a central role in care coordination. However, combining heterogeneous data streams into high-level, meaningful content - such as retrospective summaries - remains challenging. While recent work has demonstrated the promise of large language models (LLMs) for interpreting multi-modal tracking data, less attention has been given to generating narrative accounts for stakeholders like RFMs, who possess rich personal knowledge of older adults and strong emotional responsibility, yet have limited visibility into their daily lives and limited capacity for caregiving. In this work, we explore how LLMs can be used to generate retrospective summaries from multi-modal tracking data for RFMs of older adults. We leveraged and customized an existing system, Vital Insight and GLOSS, to generate initial summaries on different dates and data availability scenarios as technology probes, and conducted interviews with 11 RFMs to gather feedback. Through thematic analysis, we found that RFMs' sensemaking shift from simply presenting ''What'' data were collected, to explaining ''How'' is my loved one doing and ''Why''.