# Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models

arXiv: https://arxiv.org/abs/2305.09802 · HTML: https://mlsanity.com/paper/2305.09802/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 64 references that are in the map.
- First posted on arXiv 16 May 2023.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting](https://mlsanity.com/paper/2303.14100/) (2023)
- ["Get ready for a party": Exploring smarter smart spaces with help from large language models](https://mlsanity.com/paper/2303.14143/) (2023)

## Built on by (most cited first, 11 of 11)

- [SAGE: Smart home Agent with Grounded Execution](https://mlsanity.com/paper/2311.00772/) (2023)
- [Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors](https://mlsanity.com/paper/2410.20034/) (2024)
- [GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents](https://mlsanity.com/paper/2310.12821/) (2023)
- [Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Survey](https://mlsanity.com/paper/2412.14708/) (2024)
- [Advances in Artificial Intelligence: A Review for the Creative Industries](https://mlsanity.com/paper/2501.02725/) (2025)
- [Game of LLMs: Discovering Structural Constructs in Activities using Large Language Models](https://mlsanity.com/paper/2406.13777/) (2024)
- [Gestura: A LVLM-Powered System Bridging Motion and Semantics for Real-Time Free-Form Gesture Understanding](https://mlsanity.com/paper/2510.21814/) (2025)
- [HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network](https://mlsanity.com/paper/2601.11676/) (2026)
- [An Embodied AR Navigation Agent: Integrating BIM with Retrieval-Augmented Generation for Language Guidance](https://mlsanity.com/paper/2508.16602/) (2025)
- [MiCU: End-to-End Smart Home Command Understanding with Large Language Model](https://mlsanity.com/paper/2606.01099/) (2026)
- [AdaHome: An Adaptive Smart Home Assistant using Local Small Language Models](https://mlsanity.com/paper/2607.18034/) (2026)

## Abstract (arXiv)

Smart home assistants function best when user commands are direct and well-specified (e.g., "turn on the kitchen light"), or when a hard-coded routine specifies the response. In more natural communication, however, human speech is unconstrained, often describing goals (e.g., "make it cozy in here" or "help me save energy") rather than indicating specific target devices and actions to take on those devices. Current systems fail to understand these under-specified commands since they cannot reason about devices and settings as they relate to human situations. We introduce large language models (LLMs) to this problem space, exploring their use for controlling devices and creating automation routines in response to under-specified user commands in smart homes. We empirically study the baseline quality and failure modes of LLM-created action plans with a survey of age-diverse users. We find that LLMs can reason creatively to achieve challenging goals, but they experience patterns of failure that diminish their usefulness. We address these gaps with Sasha, a smarter smart home assistant. Sasha responds to loosely-constrained commands like "make it cozy" or "help me sleep better" by executing plans to achieve user goals, e.g., setting a mood with available devices, or devising automation routines. We implement and evaluate Sasha in a hands-on user study, showing the capabilities and limitations of LLM-driven smart homes when faced with unconstrained user-generated scenarios.