# Learning About Objects by Learning to Interact with Them

arXiv: https://arxiv.org/abs/2006.09306 · HTML: https://mlsanity.com/paper/2006.09306/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 14 of 14)

- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Sound Adversarial Audio-Visual Navigation](https://mlsanity.com/paper/2202.10910/) (2022)
- [O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning](https://mlsanity.com/paper/2106.15087/) (2021)
- [Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery](https://mlsanity.com/paper/2105.01047/) (2021)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [Ask4Help: Learning to Leverage an Expert for Embodied Tasks](https://mlsanity.com/paper/2211.09960/) (2022)
- [Shaping embodied agent behavior with activity-context priors from egocentric video](https://mlsanity.com/paper/2110.07692/) (2021)
- [Reasoning about Actions over Visual and Linguistic Modalities: A Survey](https://mlsanity.com/paper/2207.07568/) (2022)
- [IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes](https://mlsanity.com/paper/2112.05298/) (2021)
- [Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks](https://mlsanity.com/paper/2407.03967/) (2024)
- [Self-Supervised Instance Segmentation by Grasping](https://mlsanity.com/paper/2305.06305/) (2023)

## Abstract (arXiv)

Much of the remarkable progress in computer vision has been focused around fully supervised learning mechanisms relying on highly curated datasets for a variety of tasks. In contrast, humans often learn about their world with little to no external supervision. Taking inspiration from infants learning from their environment through play and interaction, we present a computational framework to discover objects and learn their physical properties along this paradigm of Learning from Interaction. Our agent, when placed within the near photo-realistic and physics-enabled AI2-THOR environment, interacts with its world and learns about objects, their geometric extents and relative masses, without any external guidance. Our experiments reveal that this agent learns efficiently and effectively; not just for objects it has interacted with before, but also for novel instances from seen categories as well as novel object categories.