# ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models

arXiv: https://arxiv.org/abs/2403.06586 · HTML: https://mlsanity.com/paper/2403.06586/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Learning for Sensor-based Activity Recognition: A Survey](https://mlsanity.com/paper/1707.03502/) (2017)
- [Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities](https://mlsanity.com/paper/2001.07416/) (2020)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [Incorporating Domain Knowledge into Deep Neural Networks](https://mlsanity.com/paper/2103.00180/) (2021)
- [Neuro-Symbolic Artificial Intelligence: Current Trends](https://mlsanity.com/paper/2105.05330/) (2021)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [Cross-Subject Transfer Learning in Human Activity Recognition Systems using Generative Adversarial Networks](https://mlsanity.com/paper/1903.12489/) (2019)
- [Text Recognition in the Wild: A Survey](https://mlsanity.com/paper/2005.03492/) (2020)
- [TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition](https://mlsanity.com/paper/2311.08245/) (2023)
- [Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning](https://mlsanity.com/paper/1912.00512/) (2019)
- [Generating Virtual On-body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition](https://mlsanity.com/paper/2305.03187/) (2023)
- [Explainable Activity Recognition for Smart Home Systems](https://mlsanity.com/paper/2105.09787/) (2021)
- [Transfer Learning in Human Activity Recognition: A Survey](https://mlsanity.com/paper/2401.10185/) (2024)
- [GG-LLM: Geometrically Grounding Large Language Models for Zero-shot Human Activity Forecasting in Human-Aware Task Planning](https://mlsanity.com/paper/2310.20034/) (2023)
- [Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPT](https://mlsanity.com/paper/2306.02140/) (2023)
- [Context-driven Active and Incremental Activity Recognition](https://mlsanity.com/paper/1906.03033/) (2019)
- [Toward Pioneering Sensors and Features Using Large Language Models in Human Activity Recognition](https://mlsanity.com/paper/2306.16017/) (2023)

## Abstract (arXiv)

Context-aware Human Activity Recognition (HAR) is a hot research area in mobile computing, and the most effective solutions in the literature are based on supervised deep learning models. However, the actual deployment of these systems is limited by the scarcity of labeled data that is required for training. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate this issue, by infusing common-sense knowledge about human activities and the contexts in which they can be performed into HAR deep learning classifiers. Existing NeSy methods for context-aware HAR rely on knowledge encoded in logic-based models (e.g., ontologies) whose design, implementation, and maintenance to capture new activities and contexts require significant human engineering efforts, technical knowledge, and domain expertise. Recent works show that pre-trained Large Language Models (LLMs) effectively encode common-sense knowledge about human activities. In this work, we propose ContextGPT: a novel prompt engineering approach to retrieve from LLMs common-sense knowledge about the relationship between human activities and the context in which they are performed. Unlike ontologies, ContextGPT requires limited human effort and expertise. An extensive evaluation carried out on two public datasets shows how a NeSy model obtained by infusing common-sense knowledge from ContextGPT is effective in data scarcity scenarios, leading to similar (and sometimes better) recognition rates than logic-based approaches with a fraction of the effort.