mlsanityData to 7 Oct 2026

Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications

arXiv 2409.17727 ↗First posted 26 Sep 2024cs.RO cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (~7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 53 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

51 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2022Visual Language Maps for Robot Navigation11 Oct
  4. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Segment Anythingcited by 8,158
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  8. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
35 more
  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  4. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  5. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  6. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  7. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  8. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  9. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  10. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  11. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  12. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  13. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  14. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  15. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  16. 2023Language-Driven Representation Learning for Roboticscited by 163
  17. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  18. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  19. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  20. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  21. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 116
  22. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
  23. 2019Antipodal Robotic Grasping using Generative Residual Convolutional Neural Networkcited by 59
  24. 2021End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGBcited by 44
  25. 2023Open-Vocabulary Affordance Detection in 3D Point Cloudscited by 41
  26. 2019Transductive Zero-Shot Learning for 3D Point Cloud Classificationcited by 39
  27. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  28. 2023Grasp-Anything: Large-scale Grasp Dataset from Foundation Modelscited by 33
  29. 2024Language-driven Grasp Detectioncited by 26
  30. 2020Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentationcited by 20
  31. 2023Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlationcited by 18
  32. 2023Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flightcited by 11
  33. 2022Machine Learning-based Framework for Optimally Solving the Analytical Inverse Kinematics for Redundant Manipulatorscited by 9
  34. 2024Lightweight Language-driven Grasp Detection using Conditional Consistency Modelcited by 9
  35. 2023Few-Shot Point Cloud Semantic Segmentation via Contrastive Self-Supervision and Multi-Resolution Attentioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuningcited by 5
  2. 2025Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agentscited by 3
  3. 2025RoboAct-CLIP: Video-Driven Pre-training of Atomic Action Understanding for Roboticscited by 1
  4. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  5. 2026ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  2. 2025CLAW: A Vision-Language-Action Framework for Weight-Aware Robotic Grasping
  3. 2025MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
  4. 2024CLIP feature-based randomized control using images and text for multiple tasks and robots
  5. 2024A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping
  6. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
  7. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Data
  8. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation