# LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation

arXiv: https://arxiv.org/abs/2501.08558 · HTML: https://mlsanity.com/paper/2501.08558/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 26 of 26)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [CoditT5: Pretraining for Source Code and Natural Language Editing](https://mlsanity.com/paper/2208.05446/) (2022)
- [SayTap: Language to Quadrupedal Locomotion](https://mlsanity.com/paper/2306.07580/) (2023)
- [Shared Autonomy with Learned Latent Actions](https://mlsanity.com/paper/2005.03210/) (2020)
- [Controlling Assistive Robots with Learned Latent Actions](https://mlsanity.com/paper/1909.09674/) (2019)
- [LILA: Language-Informed Latent Actions](https://mlsanity.com/paper/2111.03205/) (2021)
- [Generative Expressive Robot Behaviors using Large Language Models](https://mlsanity.com/paper/2401.14673/) (2024)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [Learning Latent Actions to Control Assistive Robots](https://mlsanity.com/paper/2107.02907/) (2021)
- [VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots](https://mlsanity.com/paper/2404.04066/) (2024)
- [HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaboration](https://mlsanity.com/paper/1807.11154/) (2018)
- [Independence in the Home: A Wearable Interface for a Person with Quadriplegia to Teleoperate a Mobile Manipulator](https://mlsanity.com/paper/2312.15071/) (2023)
- [CCBERT: Self-Supervised Code Change Representation Learning](https://mlsanity.com/paper/2309.15474/) (2023)
- [High-density Electromyography for Effective Gesture-based Control of Physically Assistive Mobile Manipulators](https://mlsanity.com/paper/2312.07745/) (2023)
- [Learning Latent Actions without Human Demonstrations](https://mlsanity.com/paper/2109.08801/) (2021)
- [Intelligent Mode-switching Framework for Teleoperation](https://mlsanity.com/paper/2402.06047/) (2024)

## Built on by (most cited first, 2 of 2)

- [SAPS: Shared Autonomy for Policy Steering by Blending Teleoperation with a Pretrained VLA](https://mlsanity.com/paper/2606.15568/) (2026)
- [QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents](https://mlsanity.com/paper/2511.17855/) (2025)

## Abstract (arXiv)

Teleoperating high degrees-of-freedom (DoF) robotic manipulators via low-DoF controllers like joysticks often requires frequent switching between control modes, where each mode maps controller movements to specific robot actions. Manually performing this frequent switching can make teleoperation cumbersome and inefficient. On the other hand, existing automatic mode-switching solutions, such as heuristic-based or learning-based methods, are often task-specific and lack generalizability. In this paper, we introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach that leverages Large Language Models (LLMs) to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time. The project website with supplementary materials is at https://lams-assistance.github.io/.