# Affordance Learning for End-to-End Visuomotor Robot Control

arXiv: https://arxiv.org/abs/1903.04053 · HTML: https://mlsanity.com/paper/1903.04053/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2019.

## Built on by (most cited first, 18 of 18)

- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [Open-Vocabulary Affordance Detection in 3D Point Clouds](https://mlsanity.com/paper/2303.02401/) (2023)
- [UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2506.09284/) (2025)
- [Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platforms](https://mlsanity.com/paper/2103.03697/) (2021)
- [Back to the Manifold: Recovering from Out-of-Distribution States](https://mlsanity.com/paper/2207.08673/) (2022)
- [DROPO: Sim-to-Real Transfer with Offline Domain Randomization](https://mlsanity.com/paper/2201.08434/) (2022)
- [Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning](https://mlsanity.com/paper/2207.05480/) (2022)
- [Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2303.05344/) (2023)
- [Are standard Object Segmentation models sufficient for Learning Affordance Segmentation?](https://mlsanity.com/paper/2107.02095/) (2021)
- [Egocentric zone-aware action recognition across environments](https://mlsanity.com/paper/2409.14205/) (2024)
- [Rediscovering Affordance: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2112.12886/) (2021)
- [Real2Sim or Sim2Real: Robotics Visual Insertion using Deep Reinforcement Learning and Real2Sim Policy Adaptation](https://mlsanity.com/paper/2206.02679/) (2022)
- [Learning Generalizable Agents via Saliency-Guided Features Decorrelation](https://mlsanity.com/paper/2310.05086/) (2023)
- [A Simple Approach to Continual Learning by Transferring Skill Parameters](https://mlsanity.com/paper/2110.10255/) (2021)
- [Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2204.08573/) (2022)
- [GoferBot: A Visual Guided Human-Robot Collaborative Assembly System](https://mlsanity.com/paper/2304.08840/) (2023)
- [Bayesian deep learning of affordances from RGB images](https://mlsanity.com/paper/2109.12845/) (2021)
- [Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes](https://mlsanity.com/paper/2205.04297/) (2022)

## Abstract (arXiv)

Training end-to-end deep robot policies requires a lot of domain-, task-, and hardware-specific data, which is often costly to provide. In this work, we propose to tackle this issue by employing a deep neural network with a modular architecture, consisting of separate perception, policy, and trajectory parts. Each part of the system is trained fully on synthetic data or in simulation. The data is exchanged between parts of the system as low-dimensional latent representations of affordances and trajectories. The performance is then evaluated in a zero-shot transfer scenario using Franka Panda robot arm. Results demonstrate that a low-dimensional representation of scene affordances extracted from an RGB image is sufficient to successfully train manipulator policies. We also introduce a method for affordance dataset generation, which is easily generalizable to new tasks, objects and environments, and requires no manual pixel labeling.