# P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task arXiv: https://arxiv.org/abs/2409.11279 ยท HTML: https://mlsanity.com/paper/2409.11279/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. - Builds on 26 of its 42 references that are in the map. - First posted on arXiv 17 Sep 2024. ## Placed in - Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/) ## Builds on (most cited first, 26 of 26) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020) - [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020) - [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022) - [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019) - [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022) - [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022) - [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023) - [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023) - [BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation](https://mlsanity.com/paper/2403.09227/) (2024) - [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021) - ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023) - [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022) - [Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring](https://mlsanity.com/paper/2106.03427/) (2021) - [RobotGPT: Robot Manipulation Learning from ChatGPT](https://mlsanity.com/paper/2312.01421/) (2023) - [Neuro-Symbolic Procedural Planning with Commonsense Prompting](https://mlsanity.com/paper/2206.02928/) (2022) - [One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones](https://mlsanity.com/paper/2202.07028/) (2022) - [Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments](https://mlsanity.com/paper/2010.11940/) (2020) - [Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI](https://mlsanity.com/paper/2310.01824/) (2023) - [Target-aware Abstractive Related Work Generation with Contrastive Learning](https://mlsanity.com/paper/2205.13339/) (2022) - [Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models](https://mlsanity.com/paper/2402.14245/) (2024) - [Towards Personalized Review Summarization by Modeling Historical Reviews from Customer and Product Separately](https://mlsanity.com/paper/2301.11682/) (2023) - [A Topic-aware Summarization Framework with Different Modal Side Information](https://mlsanity.com/paper/2305.11503/) (2023) - [RAP: Retrieval-Augmented Planner for Adaptive Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.18600/) (2024) - [Local Slot Attention for Vision-and-Language Navigation](https://mlsanity.com/paper/2206.08645/) (2022) ## Built on by (most cited first, 15 of 15) - [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025) - [MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents](https://mlsanity.com/paper/2410.03450/) (2024) - [Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach](https://mlsanity.com/paper/2411.14484/) (2024) - [Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verification](https://mlsanity.com/paper/2504.04578/) (2025) - [RETAIL: Towards Real-world Travel Planning for Large Language Models](https://mlsanity.com/paper/2508.15335/) (2025) - [Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skills](https://mlsanity.com/paper/2509.18597/) (2025) - [SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Models](https://mlsanity.com/paper/2411.11323/) (2024) - [ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaboration](https://mlsanity.com/paper/2509.24230/) (2025) - [Optimizing Retrieval for RAG via Reinforcement Learning](https://mlsanity.com/paper/2510.24652/) (2025) - [End-to-End LLM Flight Planning with RAG-based Memory and Multi-modal Coach Agent](https://mlsanity.com/paper/2607.06964/) (2026) - [Test-Driven Agentic Framework for Reliable Robot Controller](https://mlsanity.com/paper/2603.00455/) (2026) - [SafePilot: A Framework for Assuring LLM-enabled Cyber-Physical Systems](https://mlsanity.com/paper/2603.21523/) (2026) - [CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning](https://mlsanity.com/paper/2606.08169/) (2026) - [GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planning](https://mlsanity.com/paper/2608.07905/) (2026) - [A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding](https://mlsanity.com/paper/2610.15666/) (2026) ## Abstract (arXiv) Embodied Everyday Task is a popular task in the embodied AI community, requiring agents to make a sequence of actions based on natural language instructions and visual observations. Traditional learning-based approaches face two challenges. Firstly, natural language instructions often lack explicit task planning. Secondly, extensive training is required to equip models with knowledge of the task environment. Previous works based on Large Language Model (LLM) either suffer from poor performance due to the lack of task-specific knowledge or rely on ground truth as few-shot samples. To address the above limitations, we propose a novel approach called Progressive Retrieval Augmented Generation (P-RAG), which not only effectively leverages the powerful language processing capabilities of LLMs but also progressively accumulates task-specific knowledge without ground-truth. Compared to the conventional RAG methods, which retrieve relevant information from the database in a one-shot manner to assist generation, P-RAG introduces an iterative approach to progressively update the database. In each iteration, P-RAG retrieves the latest database and obtains historical information from the previous interaction as experiential references for the current interaction. Moreover, we also introduce a more granular retrieval scheme that not only retrieves similar tasks but also incorporates retrieval of similar situations to provide more valuable reference experiences. Extensive experiments reveal that P-RAG achieves competitive results without utilizing ground truth and can even further improve performance through self-iterations.