# LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces

arXiv: https://arxiv.org/abs/2403.19857 · HTML: https://mlsanity.com/paper/2403.19857/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 8 of 8)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)
- [Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data](https://mlsanity.com/paper/2307.14385/) (2023)
- [Penetrative AI: Making LLMs Comprehend the Physical World](https://mlsanity.com/paper/2310.09605/) (2023)
- [Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V](https://mlsanity.com/paper/2310.19061/) (2023)

## Built on by (most cited first, 28 of 28)

- [AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applications](https://mlsanity.com/paper/2503.05346/) (2025)
- [LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data](https://mlsanity.com/paper/2406.14498/) (2024)
- [SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions](https://mlsanity.com/paper/2502.02883/) (2025)
- [GPIoT: Tailoring Small Language Models for IoT Program Synthesis and Development](https://mlsanity.com/paper/2503.00686/) (2025)
- [ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild](https://mlsanity.com/paper/2512.06721/) (2025)
- [EmbedGenius: Towards Automated Software Development for Generic Embedded IoT Systems](https://mlsanity.com/paper/2412.09058/) (2024)
- [Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM](https://mlsanity.com/paper/2410.14879/) (2024)
- [Foundation Models for CPS-IoT: Opportunities and Challenges](https://mlsanity.com/paper/2501.16368/) (2025)
- [Exploring the Capabilities of LLMs for IMU-based Fine-grained Human Activity Understanding](https://mlsanity.com/paper/2504.02878/) (2025)
- [A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis](https://mlsanity.com/paper/2506.12263/) (2025)
- [Graph-Enhanced Policy Optimization in LLM Agent Training](https://mlsanity.com/paper/2510.26270/) (2025)
- [Are We There Yet? A Measurement Study of Efficiency for LLM Applications on Mobile Devices](https://mlsanity.com/paper/2504.00002/) (2025)
- [RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language](https://mlsanity.com/paper/2505.17114/) (2025)
- [Data Can Speak for Itself: Quality-guided Utilization of Wireless Synthetic Data](https://mlsanity.com/paper/2506.23174/) (2025)
- [ConSensus: Multi-Agent Collaboration for Multimodal Sensing](https://mlsanity.com/paper/2601.06453/) (2026)
- [SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing](https://mlsanity.com/paper/2410.10741/) (2024)
- [ChainStream: An LLM-based Framework for Unified Synthetic Sensing](https://mlsanity.com/paper/2412.15240/) (2024)
- [Toward Foundation Models for Online Complex Event Detection in CPS-IoT: A Case Study](https://mlsanity.com/paper/2503.12282/) (2025)
- [ThermoLLM: Thermodynamics-Aware HVAC Control with Spatial-Semantic Knowledge Graph](https://mlsanity.com/paper/2606.22911/) (2026)
- [An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding](https://mlsanity.com/paper/2505.01743/) (2025)
- [Enabling Near-realtime Remote Sensing via Satellite-Ground Collaboration of Large Vision-Language Models](https://mlsanity.com/paper/2510.24242/) (2025)
- [MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale Language Models for Multimodal Surface Sensing](https://mlsanity.com/paper/2408.07311/) (2024)
- [SPAR: Self-supervised Placement-Aware Representation Learning for Distributed Sensing](https://mlsanity.com/paper/2505.16936/) (2025)
- [Adaptive and Robust Data Poisoning Detection and Sanitization in Wearable IoT Systems using Large Language Models](https://mlsanity.com/paper/2511.02894/) (2025)
- [Authority Inversion in LLM-Mediated Ubiquitous Systems: When Models Trust Users Over Sensors](https://mlsanity.com/paper/2605.23938/) (2026)
- [From 'What' to 'How' and 'Why': Sharing LLM-Generated Retrospective Summaries of Older Adults' Passive Tracking Data with Remote Family Members](https://mlsanity.com/paper/2606.03876/) (2026)
- [CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms](https://mlsanity.com/paper/2608.28053/) (2026)
- [AirLog: Store-Level Indoor Life Logging Made Easy](https://mlsanity.com/paper/2609.31864/) (2026)

## Abstract (arXiv)

Most studies on machine learning in sensing systems focus on low-level perception tasks that process raw sensory data within a short time window. However, many practical applications, such as human routine modeling and occupancy tracking, require high-level reasoning abilities to comprehend concepts and make inferences based on long-term sensor traces. Existing machine learning-based approaches for handling such complex tasks struggle to generalize due to the limited training samples and the high dimensionality of sensor traces, necessitating the integration of human knowledge for designing first-principle models or logic reasoning methods. We pose a fundamental question: Can we harness the reasoning capabilities and world knowledge of Large Language Models (LLMs) to recognize complex events from long-term spatiotemporal sensor traces? To answer this question, we design an effective prompting framework for LLMs on high-level reasoning tasks, which can handle traces from the raw sensor data as well as the low-level perception results. We also design two strategies to enhance performance with long sensor traces, including summarization before reasoning and selective inclusion of historical traces. Our framework can be implemented in an edge-cloud setup, running small LLMs on the edge for data summarization and performing high-level reasoning on the cloud for privacy preservation. The results show that LLMSense can achieve over 80% accuracy on two high-level reasoning tasks such as dementia diagnosis with behavior traces and occupancy tracking with environmental sensor traces. This paper provides a few insights and guidelines for leveraging LLM for high-level reasoning on sensor traces and highlights several directions for future work.