# Layout Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST)

arXiv: https://arxiv.org/abs/2405.12368 · HTML: https://mlsanity.com/paper/2405.12368/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 140 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Domain Adaptation for Visual Applications: A Comprehensive Survey](https://mlsanity.com/paper/1702.05374/) (2017)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [What makes instance discrimination good for transfer learning?](https://mlsanity.com/paper/2006.06606/) (2020)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Self-training for Few-shot Transfer Across Extreme Task Differences](https://mlsanity.com/paper/2010.07734/) (2020)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition](https://mlsanity.com/paper/2311.08245/) (2023)
- [A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning](https://mlsanity.com/paper/2111.04418/) (2021)

## Built on by (most cited first, 13 of 13)

- [Past, Present, and Future of Sensor-Based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task](https://mlsanity.com/paper/2411.14452/) (2024)
- [Large Language Models are Zero-Shot Recognizers for Activities of Daily Living](https://mlsanity.com/paper/2407.01238/) (2024)
- [Leveraging Large Language Models for Explainable Activity Recognition in Smart Homes: A Critical Evaluation](https://mlsanity.com/paper/2503.16622/) (2025)
- [AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments](https://mlsanity.com/paper/2506.11773/) (2025)
- [MuRAL: A Multi-Resident Ambient Sensor Dataset Annotated with Natural Language for Activities of Daily Living](https://mlsanity.com/paper/2504.20505/) (2025)
- [HAROOD: A Benchmark for Out-of-distribution Generalization in Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2512.10807/) (2025)
- [CARE: Contrastive Alignment for ADL Recognition from Event-Triggered Sensor Streams](https://mlsanity.com/paper/2510.16988/) (2025)
- [Improving Zero-shot ADL Recognition with Large Language Models through Event-based Context and Confidence](https://mlsanity.com/paper/2601.08241/) (2026)
- [DISCOVER: Identifying Patterns of Daily Living in Human Activities from Smart Home Data](https://mlsanity.com/paper/2503.01733/) (2025)
- [ADLGen: Synthesizing Symbolic, Event-Triggered Sensor Sequences for Human Activity Modeling](https://mlsanity.com/paper/2505.17987/) (2025)
- [Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities](https://mlsanity.com/paper/2507.21964/) (2025)
- [Triple Spectral Fusion for Sensor-based Human Activity Recognition](https://mlsanity.com/paper/2605.02743/) (2026)
- [LastAct: Trajectory-Guided Latest-Activity Localization for Real-Time Smart-Home Activity Recognition](https://mlsanity.com/paper/2606.00260/) (2026)

## Abstract (arXiv)

Human activity recognition (HAR) using ambient sensors in smart homes has numerous applications for human healthcare and wellness. However, building general-purpose HAR models that can be deployed to new smart home environments requires a significant amount of annotated sensor data and training overhead. Most smart homes vary significantly in their layouts, i.e., floor plans and the specifics of sensors embedded, resulting in low generalizability of HAR models trained for specific homes. We address this limitation by introducing a novel, layout-agnostic modeling approach for HAR systems in smart homes that utilizes the transferrable representational capacity of natural language descriptions of raw sensor data. To this end, we generate Textual Descriptions Of Sensor Triggers (TDOST) that encapsulate the surrounding trigger conditions and provide cues for underlying activities to the activity recognition models. Leveraging textual embeddings, rather than raw sensor data, we create activity recognition systems that predict standard activities across homes without either (re-)training or adaptation on target homes. Through an extensive evaluation, we demonstrate the effectiveness of TDOST-based models in unseen smart homes through experiments on benchmarked CASAS datasets. Furthermore, we conduct a detailed analysis of how the individual components of our approach affect downstream activity recognition performance.