Data to 7 Oct 2026

Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning

arXiv 2009.05085 ↗First posted 10 Sep 2020cs.ROcited by 69

Not on the map. Why?

Abstract (arXiv)

Predictive models have been at the core of many robotic systems, from quadrotors to walking robots. However, it has been challenging to develop and apply such models to practical robotic manipulation due to high-dimensional sensory observations such as images. Previous approaches to learning models in the context of robotic manipulation have either learned whole image dynamics or used autoencoders to learn dynamics in a low-dimensional latent state. In this work, we introduce model-based prediction with self-supervised visual correspondence learning, and show that not only is this indeed possible, but demonstrate that these types of predictive models show compelling performance improvements over alternative methods for vision-based RL with autoencoder-type vision training. Through simulation experiments, we demonstrate that our models provide better generalization precision, particularly in 3D scenes, scenes involving occlusion, and in category-generalization. Additionally, we validate that our method effectively transfers to the real world through hardware experiments. Videos and supplementary materials available at https://sites.google.com/view/keypointsintothefuture

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 19; 2023: 15; 2024: 16; 2025: 4; 2026: 3.

69 papers.

Sort:

By area:

2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
2023Any-point Trajectory Modeling for Policy Learningcited by 207
2022Masked World Models for Visual Controlcited by 169
2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
20213D Neural Scene Representations for Visuomotor Controlcited by 110
2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
2022NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fieldscited by 86
2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79

38 more

2024Local All-Pair Correspondence for Point Trackingcited by 77
2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
2021KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Controlcited by 50
2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
2022Reinforcement Learning with Neural Radiance Fieldscited by 43
2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
2021Test-Time Personalization with a Transformer for Human Pose Estimationcited by 33
2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
2021Learning Models as Functionals of Signed-Distance Fields for Manipulation Planningcited by 29
2021VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigationcited by 27
2022K-VIL: Keypoints-based Visual Imitation Learningcited by 27
2022USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulationcited by 27
2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Controlcited by 24
2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
2023Dynamic-Resolution Model Learning for Object Pile Manipulationcited by 22
2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2021Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Taskscited by 18
2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 18
2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17
2024Composable Part-Based Manipulationcited by 17
2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
2023One-shot Imitation Learning via Interaction Warpingcited by 16
2023Model-Based Control with Sparse Neural Dynamicscited by 16
2022Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Spacecited by 14
2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignmentcited by 14
20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
2024Reinforcement Learning with Generalizable Gaussian Splattingcited by 11
2024Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptorscited by 10
2022Where To Start? Transferring Simple Skills to Complex Environmentscited by 9
2023Multi-Object Manipulation via Object-Centric Neural Scattering Functionscited by 9

50 most cited of 69 All 69 (JSON)