# LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Model

arXiv: https://arxiv.org/abs/2311.17406 · HTML: https://mlsanity.com/paper/2311.17406/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [Generalized Planning in PDDL Domains with Pretrained Large Language Models](https://mlsanity.com/paper/2305.11014/) (2023)
- [Translating Natural Language to Planning Goals with Large-Language Models](https://mlsanity.com/paper/2302.05128/) (2023)
- [Learning to Model the World with Language](https://mlsanity.com/paper/2308.01399/) (2023)
- [Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting](https://mlsanity.com/paper/2303.14100/) (2023)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds](https://mlsanity.com/paper/2305.17590/) (2023)
- [Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actors](https://mlsanity.com/paper/2307.11922/) (2023)

## Built on by (most cited first, 15 of 15)

- [AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents](https://mlsanity.com/paper/2407.04363/) (2024)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant](https://mlsanity.com/paper/2409.20548/) (2024)
- [ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models](https://mlsanity.com/paper/2505.13180/) (2025)
- [Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations](https://mlsanity.com/paper/2605.14175/) (2026)
- [Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development](https://mlsanity.com/paper/2510.12253/) (2025)
- [LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.18077/) (2026)
- [From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narratives](https://mlsanity.com/paper/2607.00918/) (2026)
- [Agentic Episodic Control](https://mlsanity.com/paper/2506.01442/) (2025)
- [Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisation](https://mlsanity.com/paper/2603.06064/) (2026)
- [World-Model Collapse as a Phase Transition](https://mlsanity.com/paper/2606.31399/) (2026)
- [No Action Without a NOD: A Heterogeneous Multi-Agent Architecture for Reliable Service Agents](https://mlsanity.com/paper/2605.12240/) (2026)
- [STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration](https://mlsanity.com/paper/2608.27225/) (2026)
- [Bridging Learned Visual Perception and Symbolic Belief-Space Planning](https://mlsanity.com/paper/2609.16884/) (2026)

## Abstract (arXiv)

This work addresses the problem of long-horizon task planning with the Large Language Model (LLM) in an open-world household environment. Existing works fail to explicitly track key objects and attributes, leading to erroneous decisions in long-horizon tasks, or rely on highly engineered state features and feedback, which is not generalizable. We propose an open state representation that provides continuous expansion and updating of object attributes from the LLM's inherent capabilities for context understanding and historical action reasoning. Our proposed representation maintains a comprehensive record of an object's attributes and changes, enabling robust retrospective summary of the sequence of actions leading to the current state. This allows continuously updating world model to enhance context understanding for decision-making in task planning. We validate our model through experiments across simulated and real-world task planning scenarios, demonstrating significant improvements over baseline methods in a variety of tasks requiring long-horizon state tracking and reasoning. (VideofootnoteVideo demonstration: urlhttps://youtu.be/QkN-8pxV3Mo.)