# LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

arXiv: https://arxiv.org/abs/2308.14972 · HTML: https://mlsanity.com/paper/2308.14972/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 5 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 2 of 2)

- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning](https://mlsanity.com/paper/2305.07716/) (2023)

## Built on by (most cited first, 17 of 17)

- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models](https://mlsanity.com/paper/2503.13107/) (2025)
- [AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models](https://mlsanity.com/paper/2406.10900/) (2024)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models](https://mlsanity.com/paper/2409.19471/) (2024)
- [The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?](https://mlsanity.com/paper/2504.10020/) (2025)
- [A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy](https://mlsanity.com/paper/2506.09420/) (2025)
- [SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense](https://mlsanity.com/paper/2510.16596/) (2025)
- [HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2506.17587/) (2025)
- [CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models](https://mlsanity.com/paper/2601.00659/) (2026)
- [Integrating Large Language Models for UAV Control in Simulated Environments: A Modular Interaction Approach](https://mlsanity.com/paper/2410.17602/) (2024)
- [NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors](https://mlsanity.com/paper/2602.22144/) (2026)
- [Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System](https://mlsanity.com/paper/2506.05020/) (2025)
- [SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision](https://mlsanity.com/paper/2508.03177/) (2025)
- [LA-RCS: LLM-Agent-Based Robot Control System](https://mlsanity.com/paper/2505.18214/) (2025)

## Abstract (arXiv)

This paper presents a novel approach to enhance autonomous robotic manipulation using the Large Language Model (LLM) for logical inference, converting high-level language commands into sequences of executable motion functions. The proposed system combines the advantage of LLM with YOLO-based environmental perception to enable robots to autonomously make reasonable decisions and task planning based on the given commands. Additionally, to address the potential inaccuracies or illogical actions arising from LLM, a combination of teleoperation and Dynamic Movement Primitives (DMP) is employed for action correction. This integration aims to improve the practicality and generalizability of the LLM-based human-robot collaboration system.