mlsanityData to 7 Oct 2026

CLIPort: What and Where Pathways for Robotic Manipulation

arXiv 2109.12098 ↗First posted 24 Sep 2021cs.RO cs.CL cs.CVcited by 617

Not on the map. Why?

Cited by 80 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

Key facts. Cited by 617 papers in the map (data through 7 Oct 2026). Builds on 49 of its 81 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

49 of 81 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017Mask R-CNNcited by 7,259
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2018SlowFast Networks for Video Recognitioncited by 1,862
37 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2016Group Equivariant Convolutional Networkscited by 1,154
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2019Solving Rubik's Cube with a Robot Handcited by 699
  8. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  9. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  10. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  11. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  12. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  13. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  14. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  15. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  16. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  17. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  18. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  19. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  20. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  21. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  22. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  23. 2019Self-supervised 6D Object Pose Estimation for Robot Manipulationcited by 100
  24. 2020Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networkscited by 77
  25. 2019Learning to Manipulate Deformable Objects without Demonstrationscited by 76
  26. 2019KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objectscited by 75
  27. 2016Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challengecited by 70
  28. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  29. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  30. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  31. 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
  32. 2020Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Datacited by 51
  33. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  34. 2019The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentationcited by 39
  35. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  36. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  37. 2017End-to-End Learning of Semantic Graspingcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 72; 2023: 151; 2024: 161; 2025: 126; 2026: 101.

617 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
38 more
  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2021Image Segmentation Using Text and Image Promptscited by 509
  4. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  5. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  6. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  8. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  9. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  13. 2023Language Models can Solve Computer Taskscited by 371
  14. 2022Visual Language Maps for Robot Navigationcited by 362
  15. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  16. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  17. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  18. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  19. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  20. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  21. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  22. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  23. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  24. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  25. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  26. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  27. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  28. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  29. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  30. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  31. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  32. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  33. 2024RT-H: Action Hierarchies Using Languagecited by 204
  34. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  35. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  36. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  37. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  38. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180

50 most cited of 617 All 617 (JSON)