# LLM Granularity for On-the-Fly Robot Control

arXiv: https://arxiv.org/abs/2406.14653 · HTML: https://mlsanity.com/paper/2406.14653/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 12 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Interactive Language: Talking to Robots in Real Time](https://mlsanity.com/paper/2210.06407/) (2022)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots](https://mlsanity.com/paper/2404.04066/) (2024)
- [Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Households](https://mlsanity.com/paper/2404.09001/) (2024)
- [How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey](https://mlsanity.com/paper/2404.00938/) (2024)
- [A Human-friendly Verbal Communication Platform for Multi-Robot Systems: Design and Principles](https://mlsanity.com/paper/2211.09519/) (2022)

## Built on by (most cited first, 2 of 2)

- [Object-Centric World Model for Language-Guided Manipulation](https://mlsanity.com/paper/2503.06170/) (2025)
- [DINOv3-Diffusion Policy: Self-Supervised Large Visual Model for Visuomotor Diffusion Policy Learning](https://mlsanity.com/paper/2509.17684/) (2025)

## Abstract (arXiv)

Assistive robots have attracted significant attention due to their potential to enhance the quality of life for vulnerable individuals like the elderly. The convergence of computer vision, large language models, and robotics has introduced the `visuolinguomotor' mode for assistive robots, where visuals and linguistics are incorporated into assistive robots to enable proactive and interactive assistance. This raises the question: textitIn circumstances where visuals become unreliable or unavailable, can we rely solely on language to control robots, i.e., the viability of the `linguomotor` mode for assistive robots? This work takes the initial steps to answer this question by: 1) evaluating the responses of assistive robots to language prompts of varying granularities; and 2) exploring the necessity and feasibility of controlling the robot on-the-fly. We have designed and conducted experiments on a Sawyer cobot to support our arguments. A Turtlebot robot case is designed to demonstrate the adaptation of the solution to scenarios where assistive robots need to maneuver to assist. Codes will be released on GitHub soon to benefit the community.