# HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices

arXiv: https://arxiv.org/abs/2505.19628 · HTML: https://mlsanity.com/paper/2505.19628/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Competitive Programming with Large Reasoning Models](https://mlsanity.com/paper/2502.06807/) (2025)
- [Towards Conversational Recommendation over Multi-Type Dialogs](https://mlsanity.com/paper/2005.03954/) (2020)
- [Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering](https://mlsanity.com/paper/2406.14891/) (2024)
- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table](https://mlsanity.com/paper/2305.07288/) (2023)
- [M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions](https://mlsanity.com/paper/2405.16420/) (2024)
- [Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models](https://mlsanity.com/paper/2404.03921/) (2024)
- [AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction](https://mlsanity.com/paper/2410.19743/) (2024)
- [IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks](https://mlsanity.com/paper/2203.12257/) (2022)
- [Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs](https://mlsanity.com/paper/2409.16341/) (2024)
- [SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systems](https://mlsanity.com/paper/2409.01630/) (2024)
- [Enhancing In-Context Learning via Implicit Demonstration Augmentation](https://mlsanity.com/paper/2407.00100/) (2024)
- [Dataset: Analysis of IFTTT Recipes to Study How Humans Use Internet-of-Things (IoT) Devices](https://mlsanity.com/paper/2110.00068/) (2021)
- [Bridging the gap between natural user expression with complex automation programming in smart homes](https://mlsanity.com/paper/2408.12687/) (2024)
- [GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents](https://mlsanity.com/paper/2310.12821/) (2023)
- [Hypergraph based Understanding for Document Semantic Entity Recognition](https://mlsanity.com/paper/2407.06904/) (2024)

## Built on by (most cited first, 15 of 15)

- [SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents](https://mlsanity.com/paper/2509.24282/) (2025)
- [PRIM: Towards Practical In-Image Multilingual Machine Translation](https://mlsanity.com/paper/2509.05146/) (2025)
- [SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios](https://mlsanity.com/paper/2511.17649/) (2025)
- [PersonalHomeBench: Evaluating Agents in Personalized Smart Homes](https://mlsanity.com/paper/2604.16813/) (2026)
- [Latent Preference Modeling for Multi-Session Personalized Tool Calling](https://mlsanity.com/paper/2604.17886/) (2026)
- [HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation](https://mlsanity.com/paper/2606.01230/) (2026)
- [Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homes](https://mlsanity.com/paper/2603.16207/) (2026)
- [DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments](https://mlsanity.com/paper/2607.13465/) (2026)
- [Trust Your Memory: Verifiable Control of Smart Homes through Reinforcement Learning with Multi-dimensional Rewards](https://mlsanity.com/paper/2604.10110/) (2026)
- [TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning](https://mlsanity.com/paper/2605.18109/) (2026)
- [SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation](https://mlsanity.com/paper/2606.22296/) (2026)
- [SimVerity: When Does Simulated Agent Success Survive Physical Deployment?](https://mlsanity.com/paper/2608.25067/) (2026)
- [Towards Generalizable Visually Grounded Exploration of Household Devices](https://mlsanity.com/paper/2609.00845/) (2026)
- [What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity](https://mlsanity.com/paper/2609.17515/) (2026)
- [OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents](https://mlsanity.com/paper/2609.32569/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have the potential to revolutionize smart home assistants by enhancing their ability to accurately understand user needs and respond appropriately, which is extremely beneficial for building a smarter home environment. While recent studies have explored integrating LLMs into smart home systems, they primarily focus on handling straightforward, valid single-device operation instructions. However, real-world scenarios are far more complex and often involve users issuing invalid instructions or controlling multiple devices simultaneously. These have two main challenges: LLMs must accurately identify and rectify errors in user instructions and execute multiple user instructions perfectly. To address these challenges and advance the development of LLM-based smart home assistants, we introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices in this paper. We have experimental results on 13 distinct LLMs; e.g., GPT-4o achieves only a 0.0% success rate in the scenario of invalid multi-device instructions, revealing that the existing state-of-the-art LLMs still cannot perform well in this situation even with the help of in-context learning, retrieval-augmented generation, and fine-tuning. Our code and dataset are publicly available at https://github.com/BITHLP/HomeBench.