# SAGE: Smart home Agent with Grounded Execution

arXiv: https://arxiv.org/abs/2311.00772 · HTML: https://mlsanity.com/paper/2311.00772/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [OpenAGI: When LLM Meets Domain Experts](https://mlsanity.com/paper/2304.04370/) (2023)
- [Lemur: Harmonizing Natural Language and Code for Language Agents](https://mlsanity.com/paper/2310.06830/) (2023)
- [Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models](https://mlsanity.com/paper/2305.09802/) (2023)
- [Dataset: Analysis of IFTTT Recipes to Study How Humans Use Internet-of-Things (IoT) Devices](https://mlsanity.com/paper/2110.00068/) (2021)
- [Evaluating Sequence-to-Sequence Learning Models for If-Then Program Synthesis](https://mlsanity.com/paper/2002.03485/) (2020)

## Built on by (most cited first, 13 of 13)

- [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026)
- [SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents](https://mlsanity.com/paper/2509.24282/) (2025)
- [Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment](https://mlsanity.com/paper/2602.03100/) (2026)
- [Context-aware LLM-based AI Agents for Human-centered Energy Management Systems in Smart Buildings](https://mlsanity.com/paper/2512.25055/) (2025)
- [SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes](https://mlsanity.com/paper/2606.01912/) (2026)
- [HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation](https://mlsanity.com/paper/2606.01230/) (2026)
- [Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Models](https://mlsanity.com/paper/2507.08878/) (2025)
- [Auto-Policy, not Auto-Skill: Compiled Agent Skills for the Physical World](https://mlsanity.com/paper/2608.25091/) (2026)
- [Safe Skill Retirement for Physical Agents](https://mlsanity.com/paper/2609.29543/) (2026)
- [Intelligence of Things: A Spatial Context-Aware Control System for Smart Devices](https://mlsanity.com/paper/2504.13942/) (2025)
- [SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation](https://mlsanity.com/paper/2606.22296/) (2026)
- [Unified Agent: Managing Interactions across Devices](https://mlsanity.com/paper/2608.05729/) (2026)
- [SimVerity: When Does Simulated Agent Success Survive Physical Deployment?](https://mlsanity.com/paper/2608.25067/) (2026)

## Abstract (arXiv)

The common sense reasoning abilities and vast general knowledge of Large Language Models (LLMs) make them a natural fit for interpreting user requests in a Smart Home assistant context. LLMs, however, lack specific knowledge about the user and their home limit their potential impact. SAGE (Smart Home Agent with Grounded Execution), overcomes these and other limitations by using a scheme in which a user request triggers an LLM-controlled sequence of discrete actions. These actions can be used to retrieve information, interact with the user, or manipulate device states. SAGE controls this process through a dynamically constructed tree of LLM prompts, which help it decide which action to take next, whether an action was successful, and when to terminate the process. The SAGE action set augments an LLM's capabilities to support some of the most critical requirements for a Smart Home assistant. These include: flexible and scalable user preference management ("is my team playing tonight?"), access to any smart device's full functionality without device-specific code via API reading "turn down the screen brightness on my dryer", persistent device state monitoring ("remind me to throw out the milk when I open the fridge"), natural device references using only a photo of the room ("turn on the light on the dresser"), and more. We introduce a benchmark of 50 new and challenging smart home tasks where SAGE achieves a 75% success rate, significantly outperforming existing LLM-enabled baselines (30% success rate).