# ExTraCT -- Explainable Trajectory Corrections from language inputs using Textual description of features

arXiv: https://arxiv.org/abs/2401.03701 · HTML: https://mlsanity.com/paper/2401.03701/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2024.

## Built on by (most cited first, 5 of 5)

- [Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors](https://mlsanity.com/paper/2402.04210/) (2024)
- [Enhancing Concept Localization in CLIP-based Concept Bottleneck Models](https://mlsanity.com/paper/2510.07115/) (2025)
- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperation](https://mlsanity.com/paper/2602.19304/) (2026)
- [Trajectory Adaptation using Large Language Models](https://mlsanity.com/paper/2504.12755/) (2025)

## Abstract (arXiv)

Natural language provides an intuitive and expressive way of conveying human intent to robots. Prior works employed end-to-end methods for learning trajectory deformations from language corrections. However, such methods do not generalize to new initial trajectories or object configurations. This work presents ExTraCT, a modular framework for trajectory corrections using natural language that combines Large Language Models (LLMs) for natural language understanding and trajectory deformation functions. Given a scene, ExTraCT generates the trajectory modification features (scene-specific and scene-independent) and their corresponding natural language textual descriptions for the objects in the scene online based on a template. We use LLMs for semantic matching of user utterances to the textual descriptions of features. Based on the feature matched, a trajectory modification function is applied to the initial trajectory, allowing generalization to unseen trajectories and object configurations. Through user studies conducted both in simulation and with a physical robot arm, we demonstrate that trajectories deformed using our method were more accurate and were preferred in about 80% of cases, outperforming the baseline. We also showcase the versatility of our system in a manipulation task and an assistive feeding task.