mlsanityData to 7 Oct 2026

VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought

arXiv 2406.14596 ↗First posted 20 Jun 2024cs.CV cs.AI cs.LGcited by 55

On the map: LLM reasoning → Multimodal reasoning

also in 3 more
Abstract (arXiv)

Large-scale generative language and vision-language models (LLMs and VLMs) excel in few-shot learning but require high-quality demonstrations. We propose In-Context Abstraction Learning (ICAL), enabling VLM agents to transform suboptimal trajectories into high-quality training data through self-reflection and human feedback. Given imperfect task demonstrations, a VLM abstracts trajectories into generalized strategies and action annotations by correcting inefficiencies and annotating cognitive abstractions: causal relationships, object state changes, temporal subgoals, and task-relevant visual elements. These annotations are iteratively refined through human feedback during execution in similar environments. The resulting examples significantly improve decision-making when used for retrieval-augmented generation or fine-tuning. As the agent's example library grows, it becomes more efficient at abstracting new examples, requiring less human feedback and fewer environment interactions. ICAL achieves state-of-the-art results across multiple benchmarks. In TEACh dialogue-based instruction following, combining fine-tuning and retrieval on ICAL examples outperforms raw human demonstrations and expert examples by 17.5% in goal-condition success. In VisualWebArena, retrieval-augmented GPT-4V with ICAL improves task success 1.6x, while fine-tuned Qwen2-VL achieves 2.8x improvement over the base model. In Ego4D action forecasting, we surpass few-shot GPT-4V and remain competitive with supervised models. Our approach scales 2x better than raw demonstrations and significantly reduces manual prompt engineering requirements.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 96 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

35 of 96 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  9. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  12. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
23 more
  1. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  2. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  3. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  4. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  5. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  6. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  7. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  8. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  9. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  10. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  11. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  12. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  13. 2023Navigating to Objects Specified by Imagescited by 50
  14. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  15. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  16. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  17. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  18. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  19. 2024Learning to Learn Faster from Human Feedback with Language Model Predictive Controlcited by 25
  20. 2024Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skillscited by 24
  21. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  22. 2017Guiding Reinforcement Learning Exploration Using Natural Languagecited by 10
  23. 2021Influencing Reinforcement Learning through Natural Language Guidancecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 27; 2026: 27.

55 papers.

Sort:

By area:

  1. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  2. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  3. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  4. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  5. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  6. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  8. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  9. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  10. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  11. 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
  12. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
38 more
  1. 2025WALT: Web Agents that Learn Toolscited by 15
  2. 2026SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scalecited by 15
  3. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  4. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
  5. 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learningcited by 9
  6. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
  7. 2025Memento No More: Coaching AI Agents to Master Multiple Tasks via Hints Internalizationcited by 8
  8. 2025Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitivescited by 7
  9. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  10. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  11. 2025Recon-Act: A Self-Evolving Multi-Agent Browser-Use System via Web Reconnaissance, Tool Generation, and Task Executioncited by 5
  12. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  13. 2025Building LLM Agents by Incorporating Insights from Computer Systemscited by 4
  14. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  15. 2025ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replaycited by 4
  16. 2026MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedbackcited by 4
  17. 2025Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languagescited by 3
  18. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  19. 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifierscited by 3
  20. 2026M2: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrievalcited by 3
  21. 2026FinEvolveBench: A Benchmark for Self-Evolving Agents on Low-Repetition Tasks with Implicit Rewardscited by 3
  22. 2026Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineeringcited by 3
  23. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  24. 2026Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Miningcited by 2
  25. 2026MemArbiter: Decision-Time Memory Arbitration for Long-Horizon LLM Agentscited by 2
  26. 2025Information Seeking for Robust Decision Making under Partial Observabilitycited by 1
  27. 2026MineNPC-Task: Task Suite for Memory-Aware Minecraft Agentscited by 1
  28. 2026OS-Marathon: Benchmarking Computer-Use Agents on Vast-Horizon, Repetitive Taskscited by 1
  29. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agentscited by 1
  30. 2026Task-Aware Scanning Parameter Configuration for Robotic Inspection Using Vision Language Embeddings and Hyperdimensional Computingcited by 1
  31. 2026PANDO: Efficient Multimodal AI Agents via Online Skill Distillationcited by 1
  32. 2026HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learningcited by 1
  33. 2026Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agentscited by 1
  34. 2025CLIPTime: Time-Aware Multimodal Representation Learning from Images and Textcited by 0
  35. 2026VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesiscited by 0
  36. 2026Inducing Reasoning Primitives from Agent Tracescited by 0
  37. 2026WebChallenger: A Reliable and Efficient Generalist Web Agentcited by 0
  38. 2026SCOPE: Evolving Symbolic World for Planning in Open-Ended Environmentscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents
  2. 2026V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments
  3. 2026In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
  4. 2025Leveraging In-Context Learning for Language Model Agents
  5. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  6. 2025EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Models
  7. 2026In-Context Robot Learning with VLM Agents
  8. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective