mlsanityData to 7 Oct 2026

Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

arXiv 2607.13653 ↗First posted 15 Jul 2026cs.CV cs.ROcited by 2

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Real-world deployment of embodied agents requires active exploration, visual grounding, and interactive intent disambiguation. However, existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges. To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent environment APIs without oracle perception and integrates a simulated user to enable human-in-the-loop interaction. Within this environment, we design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning, systematically optimizing agent performance. To comprehensively evaluate this approach, we introduce REAL-Bench, a benchmark spanning 241 tasks across active exploration, visual distraction, articulated manipulation, and interactive disambiguation. Experimental results demonstrate that our trained agent outperforms leading commercial closed-source VLMs on interactive tasks with a 56.9% success rate. Further empirical analysis reveals that our hierarchical training pipeline successfully aligns the model's tool-use capabilities while maintaining robust open-vocabulary reasoning under extended exploration horizons. Finally, we deploy and evaluate our framework on a physical dual-arm mobile robot, where it achieves a 78.3% end-to-end success rate over 60 real-world episodes. These physical trials demonstrate robust zero-shot transferability to unseen household scenarios, validating that our sim-to-real-consistent design successfully bridges the reality gap for long-horizon mobile manipulation. Code is available at https://github.com/InternRobotics/REAL.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 59 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

21 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning14 Oct
  2. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making27 May
  3. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaboration6 May
  4. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills16 Mar
  5. 2024Simulating User Agents for Embodied Conversational-AI31 Oct

Other references · 16

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  6. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  7. 2025OpenAI GPT-5 System Cardcited by 910
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2025Group Sequence Policy Optimizationcited by 788
  10. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  11. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  12. 2024RePLan: Robotic Replanning with Perception and Language Modelscited by 39
4 more
  1. 2025Communication-Efficient Desire Alignment for Embodied Agent-Human Adaptationcited by 4
  2. 2025VL-LN Bench: Towards Long-horizon Goal-oriented Navigation with Active Dialogscited by 4
  3. 2025EmboMatrix: A Scalable Training-Ground for Embodied Decision-Makingcited by 3
  4. 2025EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environmentscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Passive Execution to Active Exploration: Agentic Embodied Manipulation in Realistic Environments
  2. 2026Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation
  3. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness
  4. 2025AgentWorld: An Interactive Simulation Platform for Scene Construction and Mobile Robotic Manipulation
  5. 2026RoboBridge: A Self-Evolving Embodied Agent Framework for Sim-to-Real Transfer
  6. 2026CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
  7. 2025OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
  8. 2025ODYSSEY: Open-World Quadrupeds Exploration and Manipulation for Long-Horizon Tasks