# Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration

arXiv: https://arxiv.org/abs/2406.14097 · HTML: https://mlsanity.com/paper/2406.14097/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [ProDMPs: A Unified Perspective on Dynamic and Probabilistic Movement Primitives](https://mlsanity.com/paper/2210.01531/) (2022)
- [SoftGPT: Learn Goal-oriented Soft Object Manipulation Skills by Generative Pre-trained Heterogeneous Graph Transformer](https://mlsanity.com/paper/2306.12677/) (2023)
- [Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement](https://mlsanity.com/paper/2210.00215/) (2022)
- [NimbRo wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned](https://mlsanity.com/paper/2308.12238/) (2023)

## Built on by (most cited first, 19 of 19)

- [SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data](https://mlsanity.com/paper/2505.20347/) (2025)
- [Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model](https://mlsanity.com/paper/2409.09575/) (2024)
- [VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specifications](https://mlsanity.com/paper/2409.09536/) (2024)
- [Analytic Subspace Routing: How Recursive Least Squares Works in Continual Learning of Large Language Model](https://mlsanity.com/paper/2503.13575/) (2025)
- [ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts](https://mlsanity.com/paper/2505.10010/) (2025)
- [A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics](https://mlsanity.com/paper/2505.12583/) (2025)
- [Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System](https://mlsanity.com/paper/2506.05020/) (2025)
- [PIP-LLM: Integrating PDDL-Integer Programming with LLMs for Coordinating Multi-Robot Teams Using Natural Language](https://mlsanity.com/paper/2510.22784/) (2025)
- [CoFL: Continuous Flow Fields for Language-Conditioned Navigation](https://mlsanity.com/paper/2603.02854/) (2026)
- [LA-RCS: LLM-Agent-Based Robot Control System](https://mlsanity.com/paper/2505.18214/) (2025)
- [LLM-Driven Self-Refinement for Embodied Drone Task Planning](https://mlsanity.com/paper/2508.15501/) (2025)
- [Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction](https://mlsanity.com/paper/2602.14551/) (2026)
- [REFLEX: Metacognitive Reasoning for Reflective Zero-Shot Robotic Planning with Large Language Models](https://mlsanity.com/paper/2505.14899/) (2025)
- [GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models](https://mlsanity.com/paper/2506.11042/) (2025)
- [LLMs-guided adaptive compensator: Bringing Adaptivity to Automatic Control Systems with Large Language Models](https://mlsanity.com/paper/2507.20509/) (2025)
- [Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Driving](https://mlsanity.com/paper/2510.01126/) (2025)
- [Intent-Driven LLM Ensemble Planning for Flexible Multi-Robot Disassembly: Demonstration on EV Batteries](https://mlsanity.com/paper/2510.17576/) (2025)
- [LLM-Cave: A benchmark and light environment for large language models reasoning and decision-making system](https://mlsanity.com/paper/2511.22598/) (2025)
- [Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation](https://mlsanity.com/paper/2602.09940/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are gaining popularity in the field of robotics. However, LLM-based robots are limited to simple, repetitive motions due to the poor integration between language models, robots, and the environment. This paper proposes a novel approach to enhance the performance of LLM-based autonomous manipulation through Human-Robot Collaboration (HRC). The approach involves using a prompted GPT-4 language model to decompose high-level language commands into sequences of motions that can be executed by the robot. The system also employs a YOLO-based perception algorithm, providing visual cues to the LLM, which aids in planning feasible motions within the specific environment. Additionally, an HRC method is proposed by combining teleoperation and Dynamic Movement Primitives (DMP), allowing the LLM-based robot to learn from human guidance. Real-world experiments have been conducted using the Toyota Human Support Robot for manipulation tasks. The outcomes indicate that tasks requiring complex trajectory planning and reasoning over environments can be efficiently accomplished through the incorporation of human demonstrations.