# Grasp2Vec: Learning Object Representations from Self-Supervised Grasping

arXiv: https://arxiv.org/abs/1811.06964 · HTML: https://mlsanity.com/paper/1811.06964/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2018.

## Built on by (most cited first, 27 of 27)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Proof Artifact Co-training for Theorem Proving with Language Models](https://mlsanity.com/paper/2102.06203/) (2021)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2204.13060/) (2022)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations](https://mlsanity.com/paper/2205.06333/) (2022)
- [A Survey of Generative Techniques for Spatial-Temporal Data Mining](https://mlsanity.com/paper/2405.09592/) (2024)
- [Attribute-Based Robotic Grasping with One-Grasp Adaptation](https://mlsanity.com/paper/2104.02271/) (2021)
- [Idempotent Unsupervised Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2410.20349/) (2024)
- [Efficient Personalized Speech Enhancement through Self-Supervised Learning](https://mlsanity.com/paper/2104.02017/) (2021)
- [Learning Visual-Audio Representations for Voice-Controlled Robots](https://mlsanity.com/paper/2109.02823/) (2021)
- [Denoised Labels for Financial Time-Series Data via Self-Supervised Learning](https://mlsanity.com/paper/2112.10139/) (2021)
- [ALP: Action-Aware Embodied Learning for Perception](https://mlsanity.com/paper/2306.10190/) (2023)
- [Distribution Estimation to Automate Transformation Policies for Self-Supervision](https://mlsanity.com/paper/2111.12265/) (2021)
- [SafePicking: Learning Safe Object Extraction via Object-Level Mapping](https://mlsanity.com/paper/2202.05832/) (2022)
- [Self-Supervised Instance Segmentation by Grasping](https://mlsanity.com/paper/2305.06305/) (2023)
- [Efficient and Robust Training of Dense Object Nets for Multi-Object Robot Manipulation](https://mlsanity.com/paper/2206.12145/) (2022)
- [Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtask](https://mlsanity.com/paper/2101.06906/) (2021)
- [Cycle-Correspondence Loss: Learning Dense View-Invariant Visual Features from Unlabeled and Unordered RGB Images](https://mlsanity.com/paper/2406.12441/) (2024)
- [Efficient Hierarchical Contrastive Self-supervising Learning for Time Series Classification via Importance-aware Resolution Selection](https://mlsanity.com/paper/2502.10567/) (2025)
- [Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2605.20609/) (2026)
- [Adaptation of Quadruped Robot Locomotion with Meta-Learning](https://mlsanity.com/paper/2107.03741/) (2021)
- [Exploring Data Aggregation and Transformations to Generalize across Visual Domains](https://mlsanity.com/paper/2108.09208/) (2021)
- [Optimizing the Long-Term Behaviour of Deep Reinforcement Learning for Pushing and Grasping](https://mlsanity.com/paper/2204.03487/) (2022)
- [Grasping Student: semi-supervised learning for robotic manipulation](https://mlsanity.com/paper/2303.04452/) (2023)

## Abstract (arXiv)

Well structured visual representations can make robot learning faster and can improve generalization. In this paper, we study how we can acquire effective object-centric representations for robotic manipulation tasks without human labeling by using autonomous robot interaction with the environment. Such representation learning methods can benefit from continuous refinement of the representation as the robot collects more experience, allowing them to scale effectively without human intervention. Our representation learning approach is based on object persistence: when a robot removes an object from a scene, the representation of that scene should change according to the features of the object that was removed. We formulate an arithmetic relationship between feature vectors from this observation, and use it to learn a representation of scenes and objects that can then be used to identify object instances, localize them in the scene, and perform goal-directed grasping tasks where the robot must retrieve commanded objects from a bin. The same grasping procedure can also be used to automatically collect training data for our method, by recording images of scenes, grasping and removing an object, and recording the outcome. Our experiments demonstrate that this self-supervised approach for tasked grasping substantially outperforms direct reinforcement learning from images and prior representation learning methods.