Neurosymbolic Embodied Agents
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Language and vision-language models generate plausible embodied plans but do not guarantee executability, as their outputs can violate environment dynamics or act on incorrectly grounded entities. We present a neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning. In the first phase, a vision-language model and exploration harness acquire goal-relevant predicates and instance bindings from egocentric observations and grounded interactions, producing a symbolic initial state. In the second, a PDDL transition model restricts decoding to tokens that extend applicable actions. Monte Carlo tree search then evaluates executable continuations using a domain-independent planning heuristic. The resulting plans are executable by construction under the transition model, with transfer to the environment conditioned on correct visual grounding. On VirtualHome and ALFWorld, open 4B-27B models exceed 90% success in both environments, and our smallest agent substantially outperforms a 27B direct visual policy in each. Constraints and search prove complementary rather than interchangeable: in ALFWorld either alone solves under a third of tasks, whereas their combination solves over 95%. The method also uses several times fewer generated tokens than extended thinking and far fewer model-visible images than direct interaction, and residual failures localize to state acquisition rather than plan generation without any specialized training.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 43 references that are in the map. First posted on arXiv 17 Aug 2026.
Builds on
26 of 43 references.
Closest ancestors · 2
Same direction, newest first
- 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Models3 Oct
- 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation6 Dec
Other references · 24
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
- 2018VirtualHome: Simulating Household Activities via Programscited by 415
- 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 411
- 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
- 2020Learning to Explore using Active Neural SLAMcited by 316
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 197
- 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 101
- 2024SynCode: LLM Generation with Grammar Augmentationcited by 69
12 more
- 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 57
- 2025Flexible and Efficient Grammar-Constrained Decodingcited by 43
- 2025MiMo-Embodied: X-Embodied Foundation Model Technical Reportcited by 38
- 2025Frontier Large Language Models Rival State-of-the-Art Plannerscited by 7
- 2026XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMscited by 7
- 2025How Far Are LLMs from Symbolic Planners? An NLP-Based Perspectivecited by 6
- 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
- 2025One Demo Is All It Takes: Planning Domain Derivation with LLMs from A Single Demonstrationcited by 5
- 2025End-to-end PDDL Planning with Hardcoded and Dynamic Agentscited by 5
- 2026Agentic LLM Planning via Step-Wise PDDL Simulation: An Empirical Characterisationcited by 1
- 2026Learning and Enforcing Context-Sensitive Control for LLMscited by 1
- 2026Planning in the LLM Era: Building for Reliability and Efficiencycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Embodied Instruction Following in Unknown Environments
- 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planning
- 2026Towards Generalizable Visually Grounded Exploration of Household Devices
- 2024Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models
- 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution
- 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning
- 2026ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints
- 2024Egocentric Vision Language Planning