# VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots

arXiv: https://arxiv.org/abs/2404.04066 · HTML: https://mlsanity.com/paper/2404.04066/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 59 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder](https://mlsanity.com/paper/1711.00614/) (2017)
- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [Shared Autonomy via Hindsight Optimization for Teleoperation and Teaming](https://mlsanity.com/paper/1706.00155/) (2017)
- [Deploying and Evaluating LLMs to Program Service Mobile Robots](https://mlsanity.com/paper/2311.11183/) (2023)
- [Generative Expressive Robot Behaviors using Large Language Models](https://mlsanity.com/paper/2401.14673/) (2024)
- [Whisper: Fast Flooding for Low-Power Wireless Networks](https://mlsanity.com/paper/1809.03699/) (2018)
- [Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control](https://mlsanity.com/paper/2403.04067/) (2024)
- [Control Barrier Functions for Mechanical Systems: Theory and Application to Robotic Grasping](https://mlsanity.com/paper/1903.09816/) (2019)
- [Independence in the Home: A Wearable Interface for a Person with Quadriplegia to Teleoperate a Mobile Manipulator](https://mlsanity.com/paper/2312.15071/) (2023)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)
- [High-density Electromyography for Effective Gesture-based Control of Physically Assistive Mobile Manipulators](https://mlsanity.com/paper/2312.07745/) (2023)
- [An Adaptable, Safe, and Portable Robot-Assisted Feeding System](https://mlsanity.com/paper/2403.04134/) (2024)
- [Evaluating Customization of Remote Tele-operation Interfaces for Assistive Robots](https://mlsanity.com/paper/2304.02771/) (2023)

## Built on by (most cited first, 16 of 16)

- [Large Language Models for Medicine: A Survey](https://mlsanity.com/paper/2405.13055/) (2024)
- [FEAST: A Flexible Mealtime-Assistance System Towards In-the-Wild Personalization](https://mlsanity.com/paper/2506.14968/) (2025)
- [UWBench: A Comprehensive Vision-Language Benchmark for Underwater Understanding](https://mlsanity.com/paper/2510.18262/) (2025)
- [LLM Granularity for On-the-Fly Robot Control](https://mlsanity.com/paper/2406.14653/) (2024)
- [Towards an LLM-Based Speech Interface for Robot-Assisted Feeding](https://mlsanity.com/paper/2410.20624/) (2024)
- [Advances in Transformers for Robotic Applications: A Review](https://mlsanity.com/paper/2412.10599/) (2024)
- [LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation](https://mlsanity.com/paper/2501.08558/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)
- [Large Language Model Data Generation for Enhanced Intent Recognition in German Speech](https://mlsanity.com/paper/2508.06277/) (2025)
- [UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following](https://mlsanity.com/paper/2607.13621/) (2026)
- [Stakeholder Perspectives on Whether and How Social Robots Can Support Mediation and Advocacy for Higher Education Students with Disabilities](https://mlsanity.com/paper/2503.16499/) (2025)
- [A Social Robot with Inner Speech for Dietary Guidance](https://mlsanity.com/paper/2505.08664/) (2025)
- [KidSpeak: A General Multi-purpose LLM for Kids' Speech Recognition and Screening](https://mlsanity.com/paper/2512.05994/) (2025)
- [Robot-Assisted Social Dining as a White Glove Service](https://mlsanity.com/paper/2602.15767/) (2026)
- [Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG](https://mlsanity.com/paper/2603.09011/) (2026)
- [Exploration of Foundation Model-Based Robots in Patient and Elderly Care](https://mlsanity.com/paper/2606.10208/) (2026)

## Abstract (arXiv)

Physically assistive robots present an opportunity to significantly increase the well-being and independence of individuals with motor impairments or other forms of disability who are unable to complete activities of daily living. Speech interfaces, especially ones that utilize Large Language Models (LLMs), can enable individuals to effectively and naturally communicate high-level commands and nuanced preferences to robots. Frameworks for integrating LLMs as interfaces to robots for high level task planning and code generation have been proposed, but fail to incorporate human-centric considerations which are essential while developing assistive interfaces. In this work, we present a framework for incorporating LLMs as speech interfaces for physically assistive robots, constructed iteratively with 3 stages of testing involving a feeding robot, culminating in an evaluation with 11 older adults at an independent living facility. We use both quantitative and qualitative data from the final study to validate our framework and additionally provide design guidelines for using LLMs as speech interfaces for assistive robots. Videos and supporting files are located on our project website: https://sites.google.com/andrew.cmu.edu/voicepilot/