# LLMind: Orchestrating AI and IoT with LLM for Complex Task Execution

arXiv: https://arxiv.org/abs/2312.09007 · HTML: https://mlsanity.com/paper/2312.09007/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Deep Learning for Sentiment Analysis : A Survey](https://mlsanity.com/paper/1801.07883/) (2018)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Dynamic Planning with a LLM](https://mlsanity.com/paper/2308.06391/) (2023)
- [Towards autonomous system: flexible modular production system enhanced with large language model agents](https://mlsanity.com/paper/2304.14721/) (2023)
- [Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning](https://mlsanity.com/paper/2309.11359/) (2023)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)

## Built on by (most cited first, 24 of 24)

- [PlanGenLLMs: A Modern Survey of LLM Planning Capabilities](https://mlsanity.com/paper/2502.11221/) (2025)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Vision-Language Models for Edge Networks: A Comprehensive Survey](https://mlsanity.com/paper/2502.07855/) (2025)
- [Efficient Prompting for LLM-based Generative Internet of Things](https://mlsanity.com/paper/2406.10382/) (2024)
- [When IoT Meet LLMs: Applications and Challenges](https://mlsanity.com/paper/2411.17722/) (2024)
- [From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts](https://mlsanity.com/paper/2410.09006/) (2024)
- [ChemMiner: A Large Language Model Agent System for Chemical Literature Data Mining](https://mlsanity.com/paper/2402.12993/) (2024)
- [From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems](https://mlsanity.com/paper/2506.04565/) (2025)
- [A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis](https://mlsanity.com/paper/2506.12263/) (2025)
- [ChatIoT: Large Language Model-based Security Assistant for Internet of Things with Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.09896/) (2025)
- [Towards Pervasive Distributed Agentic Generative AI -- A State of The Art](https://mlsanity.com/paper/2506.13324/) (2025)
- [A General-Purpose Device for Interaction with LLMs](https://mlsanity.com/paper/2408.10230/) (2024)
- [Towards Edge General Intelligence via Large Language Models: Opportunities and Challenges](https://mlsanity.com/paper/2410.18125/) (2024)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [Auto-Policy, not Auto-Skill: Compiled Agent Skills for the Physical World](https://mlsanity.com/paper/2608.25091/) (2026)
- [Safe Skill Retirement for Physical Agents](https://mlsanity.com/paper/2609.29543/) (2026)
- [Rephrase and Contrast: Fine-Tuning Language Models for Enhanced Understanding of Communication and Computer Networks](https://mlsanity.com/paper/2409.19007/) (2024)
- [Integrating Semantic Communication and Human Decision-Making into an End-to-End Sensing-Decision Framework](https://mlsanity.com/paper/2412.05103/) (2024)
- [LLMs are everywhere: Ubiquitous Utilization of AI Models through Air Computing](https://mlsanity.com/paper/2503.00767/) (2025)
- [Adaptive and Robust Data Poisoning Detection and Sanitization in Wearable IoT Systems using Large Language Models](https://mlsanity.com/paper/2511.02894/) (2025)
- [Enabling Cloud-Level Accuracy in Edge AI through IoT Data Preprocessing](https://mlsanity.com/paper/2606.22496/) (2026)
- [AI Tool Discovery at Scale: All You Need is DNS](https://mlsanity.com/paper/2607.18242/) (2026)
- [Harnessing Large Language Models for Intelligent Resource Allocation in the Internet of Everything](https://mlsanity.com/paper/2607.26602/) (2026)
- [AirLog: Store-Level Indoor Life Logging Made Easy](https://mlsanity.com/paper/2609.31864/) (2026)

## Abstract (arXiv)

Task-oriented communications are an important element in future intelligent IoT systems. Existing IoT systems, however, are limited in their capacity to handle complex tasks, particularly in their interactions with humans to accomplish these tasks. In this paper, we present LLMind, an LLM-based task-oriented AI agent framework that enables effective collaboration among IoT devices, with humans communicating high-level verbal instructions, to perform complex tasks. Inspired by the functional specialization theory of the brain, our framework integrates an LLM with domain-specific AI modules, enhancing its capabilities. Complex tasks, which may involve collaborations of multiple domain-specific AI modules and IoT devices, are executed through a control script generated by the LLM using a Language-Code transformation approach, which first converts language descriptions to an intermediate finite-state machine (FSM) before final precise transformation to code. Furthermore, the framework incorporates a novel experience accumulation mechanism to enhance response speed and effectiveness, allowing the framework to evolve and become progressively sophisticated through continuing user and machine interactions.