KINet: Unsupervised Forward Models for Robotic Pushing Manipulation
Not on the map. Why?
Abstract (arXiv)
Object-centric representation is an essential abstraction for forward prediction. Most existing forward models learn this representation through extensive supervision (e.g., object class and bounding box) although such ground-truth information is not readily accessible in reality. To address this, we introduce KINet (Keypoint Interaction Network) -- an end-to-end unsupervised framework to reason about object interactions based on a keypoint representation. Using visual observations, our model learns to associate objects with keypoint coordinates and discovers a graph representation of the system as a set of keypoint embeddings and their relations. It then learns an action-conditioned forward model using contrastive estimation to predict future keypoint states. By learning to perform physical reasoning in the keypoint space, our model automatically generalizes to scenarios with a different number of objects, novel backgrounds, and unseen object geometries. Experiments demonstrate the effectiveness of our model in accurately performing forward prediction and learning plannable object-centric representations for downstream robotic pushing manipulation tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 18 Feb 2022.
Builds on
26 of 34 references.
References · 26
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
- 2018Neural Relational Inference for Interacting Systemscited by 339
- 2018Graph networks as learnable physics engines for inference and controlcited by 244
- 2019Contrastive Learning of Structured World Modelscited by 209
- 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
- 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 114
- 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
- 2017Visual Interaction Networkscited by 95
- 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
14 more
- 2018Flexible Neural Representation for Physics Predictioncited by 91
- 2020Causal Discovery in Physical Systems from Videoscited by 77
- 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
- 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
- 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 70
- 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
- 2018Propagation Networks for Model-Based Control Under Partial Observationcited by 64
- 2020Visual Grounding of Learned Physical Modelscited by 55
- 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
- 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
- 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
- 2019Structured Object-Aware Physics Prediction for Video Modeling and Planningcited by 43
- 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
- 2019Object-centric Forward Modeling for Model Predictive Controlcited by 19
Built on by
Citing papers per year: 2023: 1.
1 paper.