# Binge Watching: Scaling Affordance Learning from Sitcoms

arXiv: https://arxiv.org/abs/1804.03080 · HTML: https://mlsanity.com/paper/1804.03080/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2018.

## Built on by (most cited first, 30 of 41)

- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance](https://mlsanity.com/paper/2403.18036/) (2024)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [Making Images Real Again: A Comprehensive Survey on Deep Image Composition](https://mlsanity.com/paper/2106.14490/) (2021)
- [DECO: Dense Estimation of 3D Human-Scene Contact In The Wild](https://mlsanity.com/paper/2309.15273/) (2023)
- [Putting People in Their Place: Affordance-Aware Human Insertion into Scenes](https://mlsanity.com/paper/2304.14406/) (2023)
- [Human Hands as Probes for Interactive Object Understanding](https://mlsanity.com/paper/2112.09120/) (2021)
- [AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose](https://mlsanity.com/paper/2309.08942/) (2023)
- [One-Shot Object Affordance Detection in the Wild](https://mlsanity.com/paper/2108.03658/) (2021)
- [Generating Continual Human Motion in Diverse 3D Scenes](https://mlsanity.com/paper/2304.02061/) (2023)
- [PartAfford: Part-level Affordance Discovery from 3D Objects](https://mlsanity.com/paper/2202.13519/) (2022)
- [The One Where They Reconstructed 3D Humans and Environments in TV Shows](https://mlsanity.com/paper/2207.14279/) (2022)
- [In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data](https://mlsanity.com/paper/2511.15704/) (2025)
- [Learning Visual Affordance Grounding from Demonstration Videos](https://mlsanity.com/paper/2108.05675/) (2021)
- [TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillation](https://mlsanity.com/paper/2210.10775/) (2022)
- [The Functional Correspondence Problem](https://mlsanity.com/paper/2109.01097/) (2021)
- [Scene Synthesis from Human Motion](https://mlsanity.com/paper/2301.01424/) (2023)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom](https://mlsanity.com/paper/2404.19509/) (2024)
- [Hallucinating Pose-Compatible Scenes](https://mlsanity.com/paper/2112.06909/) (2021)
- [Visual Affordance Prediction for Guiding Robot Exploration](https://mlsanity.com/paper/2305.17783/) (2023)
- [Beyond Object Recognition: A New Benchmark towards Object Concept Learning](https://mlsanity.com/paper/2212.02710/) (2022)
- [Text2Place: Affordance-aware Text Guided Human Placement](https://mlsanity.com/paper/2407.15446/) (2024)
- [STRAP: Structured Object Affordance Segmentation with Point Supervision](https://mlsanity.com/paper/2304.08492/) (2023)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [Scene-aware Human Pose Generation using Transformer](https://mlsanity.com/paper/2308.02177/) (2023)
- [Learning Continuous Environment Fields via Implicit Functions](https://mlsanity.com/paper/2111.13997/) (2021)
- [Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion](https://mlsanity.com/paper/2412.14462/) (2024)
- [GASPACHO: Gaussian Splatting for Controllable Humans and Objects](https://mlsanity.com/paper/2503.09342/) (2025)

## Abstract (arXiv)

In recent years, there has been a renewed interest in jointly modeling perception and action. At the core of this investigation is the idea of modeling affordances(Affordances are opportunities of interaction in the scene. In other words, it represents what actions can the object be used for). However, when it comes to predicting affordances, even the state of the art approaches still do not use any ConvNets. Why is that? Unlike semantic or 3D tasks, there still does not exist any large-scale dataset for affordances. In this paper, we tackle the challenge of creating one of the biggest dataset for learning affordances. We use seven sitcoms to extract a diverse set of scenes and how actors interact with different objects in the scenes. Our dataset consists of more than 10K scenes and 28K ways humans can interact with these 10K images. We also propose a two-step approach to predict affordances in a new scene. In the first step, given a location in the scene we classify which of the 30 pose classes is the likely affordance pose. Given the pose class and the scene, we then use a Variational Autoencoder (VAE) to extract the scale and deformation of the pose. The VAE allows us to sample the distribution of possible poses at test time. Finally, we show the importance of large-scale data in learning a generalizable and robust model of affordances.