# IROSA: Interactive Robot Skill Adaptation using Natural Language

arXiv: https://arxiv.org/abs/2603.03897 · HTML: https://mlsanity.com/paper/2603.03897/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 16 of 16)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Interactive Imitation Learning in Robotics: A Survey](https://mlsanity.com/paper/2211.00600/) (2022)
- [LATTE: LAnguage Trajectory TransformEr](https://mlsanity.com/paper/2208.02918/) (2022)
- [KITE: Keypoint-Conditioned Policies for Semantic Manipulation](https://mlsanity.com/paper/2306.16605/) (2023)
- [Kernelized Movement Primitives](https://mlsanity.com/paper/1708.08638/) (2017)
- [Interactive Robot Learning from Verbal Correction](https://mlsanity.com/paper/2310.17555/) (2023)
- [Interactive incremental learning of generalizable skills with local trajectory modulation](https://mlsanity.com/paper/2409.05655/) (2024)
- [A Human-in-the-loop Approach to Robot Action Replanning through LLM Common-Sense Reasoning](https://mlsanity.com/paper/2507.20870/) (2025)
- [OVITA: Open-Vocabulary Interpretable Trajectory Adaptations](https://mlsanity.com/paper/2508.17260/) (2025)

## Built on by (most cited first, 1 of 1)

- [MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation](https://mlsanity.com/paper/2604.20468/) (2026)

## Abstract (arXiv)

Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.