# Ground Manipulator Primitive Tasks to Executable Actions using Large Language Models

arXiv: https://arxiv.org/abs/2308.06810 · HTML: https://mlsanity.com/paper/2308.06810/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 35 references that are in the map.
- First posted on arXiv 13 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Robot Behavior-Tree-Based Task Generation with Large Language Models](https://mlsanity.com/paper/2302.12927/) (2023)
- [A Motion Taxonomy for Manipulation Embedding](https://mlsanity.com/paper/2007.06695/) (2020)
- [Learning Sequences of Manipulation Primitives for Robotic Assembly](https://mlsanity.com/paper/2011.00778/) (2020)
- [Embodying Pre-Trained Word Embeddings Through Robot Actions](https://mlsanity.com/paper/2104.08521/) (2021)

## Built on by (most cited first, 1 of 1)

- [Semantic Layering in Room Segmentation via LLMs](https://mlsanity.com/paper/2403.12920/) (2024)

## Abstract (arXiv)

Layered architectures have been widely used in robot systems. The majority of them implement planning and execution functions in separate layers. However, there still lacks a straightforward way to transit high-level tasks in the planning layer to the low-level motor commands in the execution layer. In order to tackle this challenge, we propose a novel approach to ground the manipulator primitive tasks to robot low-level actions using large language models (LLMs). We designed a program-function-like prompt based on the task frame formalism. In this way, we enable LLMs to generate position/force set-points for hybrid control. Evaluations over several state-of-the-art LLMs are provided.