mlsanityData to 14 Oct 2026

D-RMGPT: Robot-assisted collaborative tasks driven by large multimodal models

arXiv 2408.11761 ↗First posted 21 Aug 2024cs.RO cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Collaborative robots are increasingly popular for assisting humans at work and daily tasks. However, designing and setting up interfaces for human-robot collaboration is challenging, requiring the integration of multiple components, from perception and robot task control to the hardware itself. Frequently, this leads to highly customized solutions that rely on large amounts of costly training data, diverging from the ideal of flexible and general interfaces that empower robots to perceive and adapt to unstructured environments where they can naturally collaborate with humans. To overcome these challenges, this paper presents the Detection-Robot Management GPT (D-RMGPT), a robot-assisted assembly planner based on Large Multimodal Models (LMM). This system can assist inexperienced operators in assembly tasks without requiring any markers or previous training. D-RMGPT is composed of DetGPT-V and R-ManGPT. DetGPT-V, based on GPT-4V(vision), perceives the surrounding environment through one-shot analysis of prompted images of the current assembly stage and the list of components to be assembled. It identifies which components have already been assembled by analysing their features and assembly requirements. R-ManGPT, based on GPT-4, plans the next component to be assembled and generates the robot's discrete actions to deliver it to the human co-worker. Experimental tests on assembling a toy aircraft demonstrated that D-RMGPT is flexible and intuitive to use, achieving an assembly success rate of 83% while reducing the assembly time for inexperienced operators by 33% compared to the manual process. http://robotics-and-ai.github.io/LMMmodels/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 50 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

39 of 50 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Segment Anythingcited by 8,234
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
27 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2021Grounded Language-Image Pre-trainingcited by 1,156
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  11. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  12. 2018Object Detection with Deep Learning: A Reviewcited by 528
  13. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  14. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  15. 2023Language Models Represent Space and Timecited by 316
  16. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  17. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  18. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  19. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 141
  20. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  21. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  22. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  23. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 43
  24. 2023RobotGPT: Robot Manipulation Learning from ChatGPTcited by 39
  25. 2023Robot Learning in the Era of Foundation Models: A Surveycited by 38
  26. 2023Large Language Models as Zero-Shot Human Models for Human-Robot Interactioncited by 30
  27. 2023Generalizable Long-Horizon Manipulations with Large Language Modelscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly
  2. 2024Towards Natural Language-Driven Assembly Using Foundation Models
  3. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  4. 2022Generalizable Human-Robot Collaborative Assembly Using Imitation Learning and Force Control
  5. 2026RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations
  6. 2025LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants
  7. 2023RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models
  8. 2024AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries