# CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision

arXiv: https://arxiv.org/abs/2411.00508 · HTML: https://mlsanity.com/paper/2411.00508/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2024.

## Built on by (most cited first, 16 of 16)

- [DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance](https://mlsanity.com/paper/2506.13922/) (2025)
- [Vision Language Action Models in Robotic Manipulation: A Systematic Review](https://mlsanity.com/paper/2507.10672/) (2025)
- [A Survey on Efficient Vision-Language-Action Models](https://mlsanity.com/paper/2510.24795/) (2025)
- [3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks](https://mlsanity.com/paper/2505.05800/) (2025)
- [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025)
- [PEAfowl: Perception-Enhanced Multi-View Vision-Language-Action for Bimanual Manipulation](https://mlsanity.com/paper/2601.17885/) (2026)
- [ChunkFlow: Towards Continuity-Consistent Chunked Policy Learning](https://mlsanity.com/paper/2607.12992/) (2026)
- [An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges](https://mlsanity.com/paper/2512.11362/) (2025)
- [EmbodiedCoder: Parameterized Embodied Mobile Manipulation via Modern Coding Model](https://mlsanity.com/paper/2510.06207/) (2025)
- [Atomic Action Slicing: Planner-Aligned Options for Generalist VLA Agents](https://mlsanity.com/paper/2512.11584/) (2025)
- [Habilis-β: A Fast-Motion and Long-Lasting On-Device Vision-Language-Action Model](https://mlsanity.com/paper/2602.18813/) (2026)
- [Grounded World Model for Semantically Generalizable Planning](https://mlsanity.com/paper/2604.11751/) (2026)
- [RoboTok: A Scalable Data Engine for Internet Demonstration Video Retrieval and Dexterous Manipulation Learning](https://mlsanity.com/paper/2609.03199/) (2026)
- [Inductive Generalization for Robotic Manipulation](https://mlsanity.com/paper/2606.20999/) (2026)
- [Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations](https://mlsanity.com/paper/2608.07895/) (2026)
- [RAIN: Robotic Task Generalization with Region-Aware Interaction Networks](https://mlsanity.com/paper/2610.14039/) (2026)

## Abstract (arXiv)

Teaching robots desired skills in real-world environments remains challenging, especially for non-experts. A key bottleneck is that collecting robotic data often requires expertise or specialized hardware, limiting accessibility and scalability. We posit that natural language offers an intuitive and accessible interface for robot learning. To this end, we study two aspects: (1) enabling non-experts to collect robotic data through natural language supervision (e.g., "move the arm to the right") and (2) training robot policies directly from this supervision. Specifically, we introduce a data collection framework that collects robot demonstrations based on natural language supervision and further augments these demonstrations. We then present CLIP-RT, a new vision-language-action (VLA) model that learns language-conditioned visuomotor policies from this supervision. CLIP-RT adapts the pretrained CLIP model and learns to predict language-based motion primitives via contrastive imitation learning. We train CLIP-RT on the Open X-Embodiment dataset and finetune it on in-domain data collected by our framework. In real-world evaluations, CLIP-RT demonstrates strong capabilities in learning novel manipulation skills, outperforming OpenVLA (7B parameters) by 24% in average success rates, while using 7x fewer parameters (1B). We further assess CLIP-RT's capabilities in few-shot generalization and collaborative scenarios involving large pretrained models or humans. In simulated environments, CLIP-RT also yields strong performance, achieving a 93.1% average success rate on the LIBERO benchmark with an inference throughput of 163 Hz.