mlsanityData to 7 Oct 2026

LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Robotics

arXiv 2504.21716 ↗First posted 30 Apr 2025cs.RO cs.AI cs.CLcited by 15

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

We present an embodied robotic system with an LLM-driven agent-orchestration architecture for autonomous household object management. The system integrates memory-augmented task planning, enabling robots to execute high-level user commands while tracking past actions. It employs three specialized agents: a routing agent, a task planning agent, and a knowledge base agent, each powered by task-specific LLMs. By leveraging in-context learning, our system avoids the need for explicit model training. RAG enables the system to retrieve context from past interactions, enhancing long-term object tracking. A combination of Grounded SAM and LLaMa3.2-Vision provides robust object detection, facilitating semantic scene understanding for task planning. Evaluation across three household scenarios demonstrates high task planning accuracy and an improvement in memory recall due to RAG. Specifically, Qwen2.5 yields best performance for specialized agents, while LLaMA3.1 excels in routing tasks. The source code is available at: https://github.com/marc1198/chat-hsr.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 26 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

21 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
8 more
  1. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  2. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  3. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  4. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  5. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  6. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  7. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  8. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 90
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  3. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  4. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  5. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Predictioncited by 25
  6. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  7. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 14
  8. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  9. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  10. 2026Evaluating Memory Structure in LLM Agentscited by 12
  11. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labelingcited by 8
  12. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 5
3 more
  1. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Queryingcited by 5
  2. 2026BrainMem: Brain-Inspired Evolving Memory for Embodied Agent Task Planningcited by 2
  3. 2026EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?
  2. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems
  3. 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution
  4. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models
  5. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning
  6. 2025MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generation
  7. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planning
  8. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments