# IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data

arXiv: https://arxiv.org/abs/2410.02429 · HTML: https://mlsanity.com/paper/2410.02429/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 31 of its 70 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs](https://mlsanity.com/paper/2402.14903/) (2024)
- [Large Language Models for Time Series: A Survey](https://mlsanity.com/paper/2402.01801/) (2024)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing](https://mlsanity.com/paper/2305.10345/) (2023)
- [Learning and Leveraging World Models in Visual Representation Learning](https://mlsanity.com/paper/2403.00504/) (2024)
- [Penetrative AI: Making LLMs Comprehend the Physical World](https://mlsanity.com/paper/2310.09605/) (2023)
- [Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence](https://mlsanity.com/paper/2306.02572/) (2023)
- [HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?](https://mlsanity.com/paper/2403.02727/) (2024)
- [The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models](https://mlsanity.com/paper/2309.06236/) (2023)

## Built on by (most cited first, 18 of 18)

- [When IoT Meet LLMs: Applications and Challenges](https://mlsanity.com/paper/2411.17722/) (2024)
- [Enhancing LLM Reasoning for Time Series Classification by Tailored Thinking and Fused Decision](https://mlsanity.com/paper/2506.00807/) (2025)
- [REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?](https://mlsanity.com/paper/2505.10872/) (2025)
- [DailyLLM: Context-Aware Activity Log Generation Using Multi-Modal Sensors and LLMs](https://mlsanity.com/paper/2507.13737/) (2025)
- [A Unified Framework for Context-Aware IoT Management and State-of-the-Art IoT Traffic Anomaly Detection](https://mlsanity.com/paper/2412.19830/) (2024)
- [ThermoLLM: Thermodynamics-Aware HVAC Control with Spatial-Semantic Knowledge Graph](https://mlsanity.com/paper/2606.22911/) (2026)
- [SignalLLM: A General-Purpose LLM Agent Framework for Automated Signal Processing](https://mlsanity.com/paper/2509.17197/) (2025)
- [Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homes](https://mlsanity.com/paper/2603.16207/) (2026)
- [Intelligence of Things: A Spatial Context-Aware Control System for Smart Devices](https://mlsanity.com/paper/2504.13942/) (2025)
- [MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning](https://mlsanity.com/paper/2505.17481/) (2025)
- [RadarPLM: Adapting Pre-trained Language Models for Marine Radar Target Detection by Selective Fine-tuning](https://mlsanity.com/paper/2509.12089/) (2025)
- [SensingAgents: A Multi-Agent Collaborative Framework for Robust IMU Activity Recognition](https://mlsanity.com/paper/2605.04608/) (2026)
- [PULSE: Agentic Investigation with Passive Sensing for Proactive Affective Intervention in Cancer Survivorship](https://mlsanity.com/paper/2605.17679/) (2026)
- [Enabling Cloud-Level Accuracy in Edge AI through IoT Data Preprocessing](https://mlsanity.com/paper/2606.22496/) (2026)
- [Design and Implementation of Agentic Orchestrations and Orchestration of Agents](https://mlsanity.com/paper/2606.31518/) (2026)
- [Persona-as-Configuration: Generative Stakeholder Reporting for Agricultural Floods](https://mlsanity.com/paper/2607.17774/) (2026)
- [Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment](https://mlsanity.com/paper/2607.20476/) (2026)
- [SenseAgent: An LLM Agent for Adaptive Cross-Domain IMU Sensing](https://mlsanity.com/paper/2609.32000/) (2026)

## Abstract (arXiv)

Large Language Models excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs knowledge via IoT-oriented retrieval-augmented generation and activating LLMs commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models like GPT-4o-mini showing a 49.4% average improvement over previous methods.