# CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learning

arXiv: https://arxiv.org/abs/2606.08169 · HTML: https://mlsanity.com/paper/2606.08169/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 6 Jun 2026.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation](https://mlsanity.com/paper/2507.05331/) (2025)
- [What Are Tools Anyway? A Survey from the Language Model Perspective](https://mlsanity.com/paper/2403.15452/) (2024)
- [Dynamic Movement Primitives in Robotics: A Tutorial Survey](https://mlsanity.com/paper/2102.03861/) (2021)
- [Kernelized Movement Primitives](https://mlsanity.com/paper/1708.08638/) (2017)
- [Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models](https://mlsanity.com/paper/2406.18746/) (2024)
- [In-Context Learning Enables Robot Action Prediction in LLMs](https://mlsanity.com/paper/2410.12782/) (2024)
- [P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task](https://mlsanity.com/paper/2409.11279/) (2024)
- [Uncertainty-Aware Imitation Learning using Kernelized Movement Primitives](https://mlsanity.com/paper/1903.02114/) (2019)
- [Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verification](https://mlsanity.com/paper/2504.04578/) (2025)
- [Learning Task-Parameterized Skills from Few Demonstrations](https://mlsanity.com/paper/2201.09975/) (2022)
- [CLEA: Closed-Loop Embodied Agent for Enhancing Task Execution in Dynamic Environments](https://mlsanity.com/paper/2503.00729/) (2025)
- [Interactive incremental learning of generalizable skills with local trajectory modulation](https://mlsanity.com/paper/2409.05655/) (2024)
- [Continual Robot Skill and Task Learning via Dialogue](https://mlsanity.com/paper/2409.03166/) (2024)

## Abstract (arXiv)

Enabling robots to understand and execute tasks from natural language commands while maintaining data efficiency remains challenging. Foundation models such as vision-language-action (VLA) and vision-language models (VLMs) provide intuitive interaction channels but require extensive data; task-parameterized imitation learning achieves data efficiency but lacks natural language grounding. This work bridges this gap through a modular architecture combining task-parameterized kernelized movement primitives (TP-KMPs) with pretrained VLMs. During learning, skills are acquired from 2 to 5 kinesthetic demonstrations, and the VLM generates skill schemas describing each skill's parameters and preconditions. During execution, the VLM interprets commands to select skills, reason about parameter bindings, and create novel behaviors through covariance-weighted composition. When no skill or composition suffices, the system identifies capability gaps and requests targeted demonstrations, all without fine-tuning. Validation on a 7-DoF manipulator shows success rates of 73.3%-100% in scenarios requiring skill selection, composition, and active learning.