# Learning Social Affordance for Human-Robot Interaction

arXiv: https://arxiv.org/abs/1604.03692 · HTML: https://mlsanity.com/paper/1604.03692/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Apr 2016.

## Built on by (most cited first, 20 of 20)

- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose](https://mlsanity.com/paper/2309.08942/) (2023)
- [SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation](https://mlsanity.com/paper/2502.13143/) (2025)
- [A Survey on Human Interaction Motion Generation](https://mlsanity.com/paper/2503.12763/) (2025)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction](https://mlsanity.com/paper/2110.00380/) (2021)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues](https://mlsanity.com/paper/2106.01006/) (2021)
- [Building Affordance Relations for Robotic Agents - A Review](https://mlsanity.com/paper/2105.06706/) (2021)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs](https://mlsanity.com/paper/2512.07410/) (2025)
- [Social Neuro AI: Social Interaction as the "dark matter" of AI](https://mlsanity.com/paper/2112.15459/) (2021)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [A Landmark-Aware Visual Navigation Dataset](https://mlsanity.com/paper/2402.14281/) (2024)
- [Triangular Character Animation Sampling with Motion, Emotion, and Relation](https://mlsanity.com/paper/2203.04930/) (2022)
- [Learning Multimodal Latent Dynamics for Human-Robot Interaction](https://mlsanity.com/paper/2311.16380/) (2023)
- [A3R: Agentic Affordance Reasoning via Cross-Dimensional Evidence in 3D Gaussian Scenes](https://mlsanity.com/paper/2604.01882/) (2026)
- [Synthetic Linguistic Agency: How an Embodied Mortal Agent Learns Linguistic Affordances through Consequential Social Experience](https://mlsanity.com/paper/2608.27843/) (2026)

## Abstract (arXiv)

In this paper, we present an approach for robot learning of social affordance from human activity videos. We consider the problem in the context of human-robot interaction: Our approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event (i.e., sub-goal). This enables the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions. We introduce the representation of social affordance and propose a generative model for its weakly supervised learning from human demonstration videos. Our approach discovers critical steps (i.e., latent sub-events) in an interaction and the typical motion associated with them, learning what body-parts should be involved and how. The experimental results demonstrate that our Markov Chain Monte Carlo (MCMC) based learning algorithm automatically discovers semantically meaningful interactive affordance from RGB-D videos, which allows us to generate appropriate full body motion for an agent.