mlsanityData to 7 Oct 2026

Visual Affordance Prediction for Guiding Robot Exploration

arXiv 2305.17783 ↗First posted 28 May 2023cs.RO cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Motivated by the intuitive understanding humans have about the space of possible interactions, and the ease with which they can generalize this understanding to previously unseen scenes, we develop an approach for learning visual affordances for guiding robot exploration. Given an input image of a scene, we infer a distribution over plausible future states that can be achieved via interactions with it. We use a Transformer-based model to learn a conditional distribution in the latent embedding space of a VQ-VAE and show that these models can be trained using large-scale and diverse passive data, and that the learned models exhibit compositional generalization to diverse objects beyond the training distribution. We show how the trained affordance model can be used for guiding exploration by acting as a goal-sampling distribution, during visual goal-conditioned policy learning in robotic manipulation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

39 of 48 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2018Exploration by Random Network Distillationcited by 966
27 more
  1. 2018Image Transformercited by 770
  2. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  3. 2019Large Scale Adversarial Representation Learningcited by 238
  4. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  5. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  6. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  7. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  8. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  9. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  10. 2018Episodic Curiosity through Reachabilitycited by 109
  11. 2020Latent Video Transformercited by 106
  12. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 80
  13. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  14. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  15. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  16. 2018Binge Watching: Scaling Affordance Learning from Sitcomscited by 41
  17. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  18. 2020Hypermodels for Explorationcited by 37
  19. 2017Characterizing and Improving Stability in Neural Style Transfercited by 31
  20. 2019Understanding the Limitations of Conditional Generative Modelscited by 28
  21. 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 22
  22. 2019Self-supervised Learning of Image Embedding for Continuous Controlcited by 16
  23. 2019Reinforcement Learning without Ground-Truth Statecited by 15
  24. 2020Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Datacited by 13
  25. 2021PixelTransformer: Sample Conditioned Signal Generationcited by 12
  26. 2020ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoningcited by 10
  27. 2018The CoSTAR Block Stacking Dataset: Learning with Workspace Constraintscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 3; 2026: 1.

12 papers.

  1. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  4. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  5. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  6. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  7. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  8. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 17
  9. 2025STAR: Learning Diverse Robot Skill Abstractions through Rotation-Augmented Vector Quantizationcited by 13
  10. 2024Seeing the Unseen: Visual Common Sense for Semantic Placementcited by 9
  11. 2025AffordanceSAM: Segment Anything Once More in Affordance Groundingcited by 6
  12. 2023How Physics and Background Attributes Impact Video Transformers in Robotic Manipulation: A Case Study on Planar Pushingcited by 0