mlsanityData to 7 Oct 2026

RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents

arXiv 2402.03610 ↗First posted 6 Feb 2024cs.LG cs.AI cs.CLcited by 80

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

Owing to recent advancements, Large Language Models (LLMs) can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming, and API integration. However, reflecting past experiences in current decision-making processes, an innate human behavior, continues to pose significant challenges. Addressing this, we propose Retrieval-Augmented Planning (RAP) framework, designed to dynamically leverage past experiences corresponding to the current situation and context, thereby enhancing agents' planning capabilities. RAP distinguishes itself by being versatile: it excels in both text-only and multimodal environments, making it suitable for a wide range of tasks. Empirical evaluations demonstrate RAP's effectiveness, where it achieves SOTA performance in textual scenarios and notably enhances multimodal LLM agents' performance for embodied tasks. These results highlight RAP's potential in advancing the functionality and applicability of LLM agents in complex, real-world applications.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 26 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

24 of 26 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
12 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  9. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  10. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  11. 2023Large Language Models as General Pattern Machinescited by 196
  12. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 33; 2026: 29.

80 papers.

Sort:

By area:

  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
  2. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  5. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  6. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  7. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
  8. 2024AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learningcited by 52
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  10. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agentscited by 45
  11. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  12. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
38 more
  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  2. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answeringcited by 38
  3. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  4. 2024THREAD: Thinking Deeper with Recursive Spawningcited by 24
  5. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  6. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 17
  7. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 17
  8. 2025ReCode: Unify Plan and Action for Universal Granularity Controlcited by 14
  9. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  10. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 12
  11. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  12. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  13. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Frameworkcited by 12
  14. 2024MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agentscited by 11
  15. 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systemscited by 10
  16. 2025Toward Agentic AI: Generative Information Retrieval Inspired Intelligent Communications and Networkingcited by 9
  17. 2025Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verificationcited by 9
  18. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9
  19. 2025InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planningcited by 8
  20. 2025H2R: Hierarchical Hindsight Reflection for Multi-Task LLM Agentscited by 8
  21. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  22. 2025AI Awarenesscited by 6
  23. 2026Retrieval-Augmented LLM Agents: Learning to Learn from Experiencecited by 6
  24. 2024Fleet of Agents: Coordinated Problem Solving with Large Language Modelscited by 5
  25. 2025EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVercited by 5
  26. 2025NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoningcited by 5
  27. 2025Cache Mechanism for Agent RAG Systemscited by 5
  28. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMscited by 4
  29. 2024Artificial Intelligence as the New Hacker: Developing Agents for Offensive Securitycited by 4
  30. 2026State Contamination in Memory-Augmented LLM Agentscited by 4
  31. 2026Learning to Retrieve: Dual-Level Long-Term Memory for Text-to-SQL Agentscited by 4
  32. 2026FragFuse: Bypassing Access Control of Large Language Model Agents via Memory-Based Query Fragmentation and Fusioncited by 4
  33. 2026Your Agent's Memories Are Not Its Own: Forged Reasoning Attacks on LLM Agent Memory and Defensescited by 4
  34. 2024DAVIS: Planning Agent with Knowledge Graph-Powered Inner Monologuecited by 3
  35. 2024SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Modelscited by 3
  36. 2025ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learningcited by 3
  37. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  38. 2026Hippocampus: An Efficient and Scalable Memory Module for Agentic AIcited by 3

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agents
  2. 2026AdMem: Advanced Memory for Task-solving Agents
  3. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
  4. 2025AUGUSTUS: An LLM-Driven Multimodal Agent System with Contextualized User Memory
  5. 2025From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
  6. 2025Coarse-to-Fine Grounded Memory for LLM Agent Planning
  7. 2024Large Multimodal Agents: A Survey
  8. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics