mlsanityData to 14 Oct 2026

LaGEA: Language Guided Embodied Agents for Robotic Manipulation

arXiv 2509.23155 ↗First posted 27 Sep 2025cs.ROcited by 2

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Robotic manipulation benefits from foundation models that describe goals, but today's agents still lack a principled way to learn from their own mistakes. We ask whether natural language can serve as feedback, an error-reasoning signal that helps embodied agents diagnose what went wrong and correct course. We introduce LaGEA (Language Guided Embodied Agents), a framework that turns episodic, schema-constrained reflections from a vision language model (VLM) into temporally grounded guidance for reinforcement learning. LaGEA summarizes each attempt in concise language, localizes the decisive moments in the trajectory, aligns feedback with visual state in a shared representation, and converts goal progress and feedback agreement into bounded, step-wise shaping rewards whose influence is modulated by an adaptive, failure-aware coefficient. This design yields dense signals early when exploration needs direction and gracefully recedes as competence grows. On the Meta-World MT10 and Robotic Fetch embodied manipulation benchmark, LaGEA improves average success over the state-of-the-art (SOTA) methods by 9.0% on random goals, 5.3% on fixed goals, and 17% on fetch tasks, while converging faster. These results support our hypothesis: language, when structured and grounded in time, is an effective mechanism for teaching robots to self-reflect on mistakes and make better choices.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 68 of its 78 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

68 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 67

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  15. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  16. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
  17. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  18. 2022Visual Language Maps for Robot Navigationcited by 365
  19. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  20. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  21. 2024What matters when building vision-language models?cited by 326
  22. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  23. 2023Reward Design with Language Modelscited by 261
  24. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  25. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  26. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  27. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  28. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  29. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  30. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  31. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  32. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  33. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 159
  34. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  35. 2023Vision-Language Models as Success Detectorscited by 142
  36. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  37. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
  38. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025OMP: One-step Meanflow Policy with Directional Alignmentcited by 4
  2. 2026What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforced Embodied Planning with Verifiable Reward for Real-World Robotic Manipulation
  2. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness
  3. 2026Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models
  4. 2023ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks
  5. 2026RoboHarness: A Simple Harness Outperforms VLA and World Actions Models
  6. 2026Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation
  7. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
  8. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation