mlsanityData to 7 Oct 2026

Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation

arXiv 2406.09738 ↗First posted 14 Jun 2024cs.RO cs.CVcited by 15

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Developing robots capable of executing various manipulation tasks, guided by natural language instructions and visual observations of intricate real-world environments, remains a significant challenge in robotics. Such robot agents need to understand linguistic commands and distinguish between the requirements of different tasks. In this work, we present Sigma-Agent, an end-to-end imitation learning agent for multi-task robotic manipulation. Sigma-Agent incorporates contrastive Imitation Learning (contrastive IL) modules to strengthen vision-language and current-future representations. An effective and efficient multi-view querying Transformer (MVQ-Former) for aggregating representative semantic information is introduced. Sigma-Agent shows substantial improvement over state-of-the-art methods under diverse settings in 18 RLBench tasks, surpassing RVT by an average of 5.2% and 5.9% in 10 and 100 demonstration training, respectively. Sigma-Agent also achieves 62% success rate with a single policy in 5 real-world manipulation tasks. The code will be released upon acceptance.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 60 of its 68 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

60 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training30 Sep
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  13. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  14. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  15. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  16. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  17. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  18. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  19. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  20. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  21. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  22. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  23. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  24. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  25. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  26. 2021Learning One Representation to Optimize All Rewardscited by 110
  27. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  28. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  29. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  30. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  31. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  32. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  33. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  34. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  35. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  36. 2019Routing Networks and the Challenges of Modular and Compositional Computationcited by 49
  37. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 49
  38. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 43

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  2. 2024ET-SEED: Efficient Trajectory-Level SE(3) Equivariant Diffusion Policycited by 24
  3. 2024Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulationcited by 22
  4. 2024GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Graspingcited by 15
  5. 2025Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulationcited by 10
  6. 2026MotionWAM: Towards Foundation World Action Models for Real-Time Humanoid Loco-Manipulationcited by 8
  7. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  8. 2025Language-Conditioned Representations and Mixture-of-Experts Policy for Robust Multi-Task Robotic Manipulationcited by 2
  9. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  10. 2026FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptationcited by 2
  11. 2026Improving Vision-Language-Action Model Fine-Tuning with Structured Stage and Keyframe Supervisioncited by 2
  12. 2025GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learningcited by 1
3 more
  1. 2025FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesiscited by 1
  2. 2026Grounding Hierarchical Vision-Language-Action Models Through Explicit Language-Action Alignmentcited by 1
  3. 2026Vision-Language Grounded Task-Context-Aware Imitation Learning for Robotic Disassemblycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
  2. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data
  3. 2025Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation
  4. 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
  5. 2026Multi-Granularity Language-Guided Imitation Learning via Instruction Decomposition
  6. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
  7. 2024Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following
  8. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks