# SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

arXiv: https://arxiv.org/abs/2606.01912 · HTML: https://mlsanity.com/paper/2606.01912/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 10 of 10)

- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments](https://mlsanity.com/paper/2407.18550/) (2024)
- [SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents](https://mlsanity.com/paper/2509.24282/) (2025)
- [SAGE: Smart home Agent with Grounded Execution](https://mlsanity.com/paper/2311.00772/) (2023)
- [Bridging the gap between natural user expression with complex automation programming in smart homes](https://mlsanity.com/paper/2408.12687/) (2024)
- [Leveraging LLMs for Efficient and Personalized Smart Home Automation](https://mlsanity.com/paper/2601.04680/) (2026)
- [Harmony: A Human-Aware, Responsive, Modular Assistant with a Locally Deployed Large Language Model](https://mlsanity.com/paper/2410.14252/) (2024)
- [SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts](https://mlsanity.com/paper/2603.06636/) (2026)
- [Towards Privacy-Preserving and Personalized Smart Homes via Tailored Small Language Models](https://mlsanity.com/paper/2507.08878/) (2025)

## Built on by (most cited first, 4 of 4)

- [DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments](https://mlsanity.com/paper/2607.13465/) (2026)
- [SimVerity: When Does Simulated Agent Success Survive Physical Deployment?](https://mlsanity.com/paper/2608.25067/) (2026)
- [What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity](https://mlsanity.com/paper/2609.17515/) (2026)
- [OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents](https://mlsanity.com/paper/2609.32569/) (2026)

## Abstract (arXiv)

Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions. However, existing smart-home benchmarks often focus on static instruction-to-API mapping or limited simulations, failing to evaluate whether LLMs can reason, interact, and act reliably in realistic household scenarios. To address these limitations, we introduce SMH-Bench, a comprehensive benchmark for evaluating LLMs in smart-home environments. Built upon HomeEnv, an executable and verifiable smart-home simulator, SMH-Bench contains 1,100 high-quality tasks spanning 7 categories and 22 fine-grained subcategories. It further stratifies tasks across simple, medium and complex homes, ranging from small apartments to dense multi-room environments with 135 devices. Experiments show that although frontier LLMs achieve strong performance on explicit control and query tasks, they still exhibit significant weaknesses in automation task scheduling, ambiguity handling and personalized reasoning, especially as home complexity increases. We hope SMH-Bench will facilitate the development of more reliable, context-aware, and practically deployable smart-home agents.