mlsanityData to 14 Oct 2026

Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learning

arXiv 2504.00907 ↗First posted 1 Apr 2025cs.AIcited by 21

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Embodied agents operating in household environments must interpret ambiguous and under-specified human instructions. A capable household robot should recognize ambiguity and ask relevant clarification questions to infer the user intent accurately, leading to more effective task execution. To study this problem, we introduce the Ask-to-Act task, where an embodied agent is tasked with a single or multi-object rearrangement task using an under-specified instruction in a home environment. The agent must strategically ask minimal, yet relevant, clarification questions to resolve ambiguity while navigating under partial observability. To address this challenge, we propose a novel approach that fine-tunes multi-modal large language models (MLLMs) as vision-language-action (VLA) policies using online reinforcement learning (RL) with LLM-generated rewards. Our method eliminates the need for large-scale human demonstrations or manually engineered rewards for training such agents. We benchmark against strong zero-shot baselines including GPT-4o as well as supervised fine-tuned MLLMs on our task. Our results show that our RL-finetuned MLLM outperforms all baselines by a significant margin (10.4-16.5%), generalizing well to novel scenes and tasks. To the best of our knowledge, this is the first demonstration of adapting MLLMs as VLA agents that can act and ask for help using LLM-generated rewards with online RL.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 58 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

52 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons11 Dec
  2. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
  3. 2024Grounding Multimodal Large Language Models in Actions12 Jun
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
36 more
  1. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  2. 2018Learning Dexterous In-Hand Manipulationcited by 687
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  4. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  6. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  7. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  8. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  9. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  10. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  11. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  12. 2019Vision-and-Dialog Navigationcited by 250
  13. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  14. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  15. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  16. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  17. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  18. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  19. 2021Visual Room Rearrangementcited by 102
  20. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  21. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  22. 2023Embodied Task Planning with Large Language Modelscited by 84
  23. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 66
  24. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  25. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  26. 2025Flexible and Efficient Grammar-Constrained Decodingcited by 43
  27. 2020The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigationcited by 26
  28. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  29. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  30. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  31. 2023Skill Transformer: A Monolithic Policy for Mobile Manipulationcited by 16
  32. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  33. 2023Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Secondcited by 5
  34. 2023Learning Reward for Physical Skills using Large Language Modelcited by 5
  35. 2024Automated Rewards via LLM-Generated Progress Functionscited by 4
  36. 2024Situated Instruction Followingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025Reinforcement Learning for Flow-Matching Policiescited by 15
  2. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 14
  3. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  4. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 10
  5. 2026Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoringcited by 9
  6. 2026Modeling Distinct Human Interaction in Web Agentscited by 5
  7. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Makingcited by 4
  8. 2025Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialoguecited by 4
  9. 2025PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environmentscited by 4
  10. 2026Can RL Improve Generalization of LLM Agents? An Empirical Studycited by 4
  11. 2025Training LLM Agents to Empower Humanscited by 3
  12. 2026Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AIcited by 3
9 more
  1. 2025ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learningcited by 2
  2. 2026When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Makingcited by 2
  3. 2025Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debatecited by 1
  4. 2025Benchmarking In-context Experiential Learning Through Repeated Product Recommendationscited by 1
  5. 2026Seeing Isn't Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agentscited by 1
  6. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  7. 2026Goal-oriented Navigation Instruction Generation with Tour Video Priorscited by 0
  8. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
  9. 2026WMLLM: Self-Evolving Optimization Agents via Predict-Then-Act World Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?
  2. 2026RoboHarness: A Simple Harness Outperforms VLA and World Actions Models
  3. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  4. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents
  5. 2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Models
  6. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction
  7. 2025HELP: Hierarchical Embodied Language Planner for Household Tasks
  8. 2025LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning