# What Can I Do Here? Learning New Skills by Imagining Visual Affordances

arXiv: https://arxiv.org/abs/2106.00671 · HTML: https://mlsanity.com/paper/2106.00671/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models](https://mlsanity.com/paper/1507.00814/) (2015)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [Unsupervised Control Through Non-Parametric Discriminative Rewards](https://mlsanity.com/paper/1811.11359/) (2018)
- [Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots](https://mlsanity.com/paper/1301.4862/) (2013)
- [Deep Affordance Foresight: Planning Through What Can Be Done in the Future](https://mlsanity.com/paper/2011.08424/) (2020)
- [Contextual Imagined Goals for Self-Supervised Robotic Learning](https://mlsanity.com/paper/1910.11670/) (2019)
- [What can I do here? A Theory of Affordances in Reinforcement Learning](https://mlsanity.com/paper/2006.15085/) (2020)
- [Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration](https://mlsanity.com/paper/1803.00781/) (2018)

## Built on by (most cited first, 23 of 23)

- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting](https://mlsanity.com/paper/2403.03174/) (2024)
- [A Workflow for Offline Model-Free Robotic Reinforcement Learning](https://mlsanity.com/paper/2109.10813/) (2021)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2204.13060/) (2022)
- [Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space](https://mlsanity.com/paper/2205.08129/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks](https://mlsanity.com/paper/2210.06601/) (2022)
- [Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learning](https://mlsanity.com/paper/2107.03961/) (2021)
- [Visual Affordance Prediction for Guiding Robot Exploration](https://mlsanity.com/paper/2305.17783/) (2023)
- [Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision](https://mlsanity.com/paper/2210.15206/) (2022)
- [Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.03172/) (2025)
- [Goal-Conditioned Reinforcement Learning with Disentanglement-based Reachability Planning](https://mlsanity.com/paper/2307.10846/) (2023)
- [Possibility Before Utility: Learning And Using Hierarchical Affordances](https://mlsanity.com/paper/2203.12686/) (2022)
- [Information-driven Affordance Discovery for Efficient Robotic Manipulation](https://mlsanity.com/paper/2405.03865/) (2024)
- [Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2202.13624/) (2022)
- [Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation](https://mlsanity.com/paper/2304.08703/) (2023)
- [Manipulate-to-Navigate: Reinforcement Learning with Visual Affordances and Manipulability Priors](https://mlsanity.com/paper/2508.13151/) (2025)
- [Hierarchical Reinforcement Learning in Complex 3D Environments](https://mlsanity.com/paper/2302.14451/) (2023)
- [The Interpretability of Codebooks in Model-Based Reinforcement Learning is Limited](https://mlsanity.com/paper/2407.19532/) (2024)
- [Proposing Hierarchical Goal-Conditioned Policy Planning in Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/2501.01727/) (2025)
- [A representational framework for learning and encoding structurally enriched trajectories in complex agent environments](https://mlsanity.com/paper/2503.13194/) (2025)

## Abstract (arXiv)

A generalist robot equipped with learned skills must be able to perform many tasks in many different environments. However, zero-shot generalization to new settings is not always possible. When the robot encounters a new environment or object, it may need to finetune some of its previously learned skills to accommodate this change. But crucially, previously learned behaviors and models should still be suitable to accelerate this relearning. In this paper, we aim to study how generative models of possible outcomes can allow a robot to learn visual representations of affordances, so that the robot can sample potentially possible outcomes in new situations, and then further train its policy to achieve those outcomes. In effect, prior data is used to learn what kinds of outcomes may be possible, such that when the robot encounters an unfamiliar setting, it can sample potential outcomes from its model, attempt to reach them, and thereby update both its skills and its outcome model. This approach, visuomotor affordance learning (VAL), can be used to train goal-conditioned policies that operate on raw image inputs, and can rapidly learn to manipulate new objects via our proposed affordance-directed exploration scheme. We show that VAL can utilize prior data to solve real-world tasks such drawer opening, grasping, and placing objects in new scenes with only five minutes of online experience in the new scene.