# Cross-Embodied Affordance Transfer through Learning Affordance Equivalences

arXiv: https://arxiv.org/abs/2404.15648 · HTML: https://mlsanity.com/paper/2404.15648/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model](https://mlsanity.com/paper/1610.03518/) (2016)
- [Deep Imitation Learning for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2010.05134/) (2020)
- [Hardware Conditioned Policies for Multi-Robot Transfer Learning](https://mlsanity.com/paper/1811.09864/) (2018)
- [Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency](https://mlsanity.com/paper/2012.09811/) (2020)
- [Kernelized Movement Primitives](https://mlsanity.com/paper/1708.08638/) (2017)
- [World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges](https://mlsanity.com/paper/2301.05832/) (2023)
- [Polybot: Training One Policy Across Robots While Embracing Variability](https://mlsanity.com/paper/2307.03719/) (2023)
- [Multimodal representation models for prediction and control from partial information](https://mlsanity.com/paper/1910.03854/) (2019)
- [Imitation and Mirror Systems in Robots through Deep Modality Blending Networks](https://mlsanity.com/paper/2106.08422/) (2021)
- [Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2303.05344/) (2023)
- [Beyond the Self: Using Grounded Affordances to Interpret and Describe Others' Actions](https://mlsanity.com/paper/1902.09705/) (2019)
- [Multi-step planning with learned effects of partial action executions](https://mlsanity.com/paper/2303.09355/) (2023)
- [Correspondence learning between morphologically different robots via task demonstrations](https://mlsanity.com/paper/2310.13458/) (2023)

## Abstract (arXiv)

Affordances represent the inherent effect and action possibilities that objects offer to the agents within a given context. From a theoretical viewpoint, affordances bridge the gap between effect and action, providing a functional understanding of the connections between the actions of an agent and its environment in terms of the effects it can cause. In this study, we propose a deep neural network model that unifies objects, actions, and effects into a single latent vector in a common latent space that we call the affordance space. Using the affordance space, our system can generate effect trajectories when action and object are given and can generate action trajectories when effect trajectories and objects are given. Our model does not learn the behavior of individual objects acted upon by a single agent. Still, rather, it forms a `shared affordance representation' spanning multiple agents and objects, which we call Affordance Equivalence. Affordance Equivalence facilitates not only action generalization over objects but also Cross Embodiment transfer linking actions of different robots. In addition to the simulation experiments that demonstrate the proposed model's range of capabilities, we also showcase that our model can be used for direct imitation in real-world settings.