# Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming

arXiv: https://arxiv.org/abs/2312.07214 · HTML: https://mlsanity.com/paper/2312.07214/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 80 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues](https://mlsanity.com/paper/2205.07540/) (2022)
- [Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architectures](https://mlsanity.com/paper/2310.03659/) (2023)
- [Towards Designing a ChatGPT Conversational Companion for Elderly People](https://mlsanity.com/paper/2304.09866/) (2023)
- [Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models](https://mlsanity.com/paper/2308.16529/) (2023)
- [A Review of Methodologies for Natural-Language-Facilitated Human-Robot Cooperation](https://mlsanity.com/paper/1701.08756/) (2017)
- [How to Communicate Robot Motion Intent: A Scoping Review](https://mlsanity.com/paper/2303.00362/) (2023)
- [In Time and Space: Towards Usable Adaptive Control for Assistive Robotic Arms](https://mlsanity.com/paper/2307.02933/) (2023)
- [Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?](https://mlsanity.com/paper/2308.06032/) (2023)
- [AdaptiX -- A Transitional XR Framework for Development and Evaluation of Shared Control Applications in Assistive Robotics](https://mlsanity.com/paper/2310.15887/) (2023)

## Built on by (most cited first, 3 of 3)

- [Large Language Models for Multi-Robot Systems: A Survey](https://mlsanity.com/paper/2502.03814/) (2025)
- [Intent-Handover: Grounding Language in Human-Usage Regions for Trustworthy Robot-to-Human Handovers](https://mlsanity.com/paper/2503.03579/) (2025)
- [Intent-Driven LLM Ensemble Planning for Flexible Multi-Robot Disassembly: Demonstration on EV Batteries](https://mlsanity.com/paper/2510.17576/) (2025)

## Abstract (arXiv)

In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with robot agents through natural language, each powered by individual GPT cores. By means of OpenAI's function calling, we bridge the gap between unstructured natural language input and structure robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their robot collaborators. Still, those users who did explore where able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems.