# Learning Action Maps of Large Environments via First-Person Vision

arXiv: https://arxiv.org/abs/1605.01679 · HTML: https://mlsanity.com/paper/1605.01679/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2016.

## Built on by (most cited first, 11 of 11)

- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [HRP: Human Affordances for Robotic Pre-Training](https://mlsanity.com/paper/2407.18911/) (2024)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Multi-label affordance mapping from egocentric vision](https://mlsanity.com/paper/2309.02120/) (2023)
- [AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2406.01194/) (2024)
- [IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes](https://mlsanity.com/paper/2112.05298/) (2021)
- [Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos](https://mlsanity.com/paper/2410.16259/) (2024)
- [Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2602.14837/) (2026)
- [Egocentric Activity Recognition and Localization on a 3D Map](https://mlsanity.com/paper/2105.09544/) (2021)
- [AROS: Affordance Recognition with One-Shot Human Stances](https://mlsanity.com/paper/2210.11725/) (2022)
- [DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videos](https://mlsanity.com/paper/2503.08344/) (2025)

## Abstract (arXiv)

When people observe and interact with physical spaces, they are able to associate functionality to regions in the environment. Our goal is to automate dense functional understanding of large spaces by leveraging sparse activity demonstrations recorded from an ego-centric viewpoint. The method we describe enables functionality estimation in large scenes where people have behaved, as well as novel scenes where no behaviors are observed. Our method learns and predicts "Action Maps", which encode the ability for a user to perform activities at various locations. With the usage of an egocentric camera to observe human activities, our method scales with the size of the scene without the need for mounting multiple static surveillance cameras and is well-suited to the task of observing activities up-close. We demonstrate that by capturing appearance-based attributes of the environment and associating these attributes with activity demonstrations, our proposed mathematical framework allows for the prediction of Action Maps in new environments. Additionally, we offer a preliminary glance of the applicability of Action Maps by demonstrating a proof-of-concept application in which they are used in concert with activity detections to perform localization.