mlsanityData to 14 Oct 2026

EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Execution

arXiv 2408.17379 ↗First posted 30 Aug 2024cs.RO cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Task planning for robots in real-life settings presents significant challenges. These challenges stem from three primary issues: the difficulty in identifying grounded sequences of steps to achieve a goal; the lack of a standardized mapping between high-level actions and low-level commands; and the challenge of maintaining low computational overhead given the limited resources of robotic hardware. We introduce EMPOWER, a framework designed for open-vocabulary online grounding and planning for embodied agents aimed at addressing these issues. By leveraging efficient pre-trained foundation models and a multi-role mechanism, EMPOWER demonstrates notable improvements in grounded planning and execution. Quantitative results highlight the effectiveness of our approach, achieving an average success rate of 0.73 across six different real-life scenarios using a TIAGo robot.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
27 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,156
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  5. 2020Knowledge Graphscited by 621
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  7. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  8. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  12. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  13. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  14. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  15. 2023Role-Play with Large Language Modelscited by 294
  16. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  17. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  18. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  19. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  20. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 89
  21. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  22. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 35
  23. 2023Large Language Models as Zero-Shot Human Models for Human-Robot Interactioncited by 30
  24. 2024LLM Based Multi-Agent Generation of Semi-structured Documents from Semantic Templates in the Public Administration Domaincited by 9
  25. 2024Multi-Agent Planning Using Visual Language Modelscited by 7
  26. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Languagecited by 4
  27. 2024Automatic Robotic Development through Collaborative Framework by Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planningcited by 3
  2. 2026LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environmentscited by 2
  3. 2025Curriculum-Based Multi-Tier Semantic Exploration via Deep Reinforcement Learningcited by 0