# You-Do, I-Learn: Unsupervised Multi-User egocentric Approach Towards Video-Based Guidance

arXiv: https://arxiv.org/abs/1510.04862 · HTML: https://mlsanity.com/paper/1510.04862/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2015.

## Built on by (most cited first, 17 of 17)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [EgoTaskQA: Understanding Human Tasks in Egocentric Videos](https://mlsanity.com/paper/2210.03929/) (2022)
- [Human Hands as Probes for Interactive Object Understanding](https://mlsanity.com/paper/2112.09120/) (2021)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Visual Object Tracking in First Person Vision](https://mlsanity.com/paper/2209.13502/) (2022)
- [EgoEnv: Human-centric environment representations from egocentric video](https://mlsanity.com/paper/2207.11365/) (2022)
- [Is First Person Vision Challenging for Object Tracking?](https://mlsanity.com/paper/2108.13665/) (2021)
- [Multi-label affordance mapping from egocentric vision](https://mlsanity.com/paper/2309.02120/) (2023)
- [A Survey of Video-based Action Quality Assessment](https://mlsanity.com/paper/2204.09271/) (2022)
- [EgoPet: Egomotion and Interaction Data from an Animal's Perspective](https://mlsanity.com/paper/2404.09991/) (2024)
- [ARDuP: Active Region Video Diffusion for Universal Policies](https://mlsanity.com/paper/2406.13301/) (2024)
- [Anticipating Next Active Objects for Egocentric Videos](https://mlsanity.com/paper/2302.06358/) (2023)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [Learning Multiple Object States from Actions via Large Language Models](https://mlsanity.com/paper/2405.01090/) (2024)
- [EgoCOL: Egocentric Camera pose estimation for Open-world 3D object Localization @Ego4D challenge 2023](https://mlsanity.com/paper/2306.16606/) (2023)
- [EgoVSR: Towards High-Quality Egocentric Video Super-Resolution](https://mlsanity.com/paper/2305.14708/) (2023)
- [PhilHumans: Benchmarking Machine Learning for Personal Health](https://mlsanity.com/paper/2405.02770/) (2024)

## Abstract (arXiv)

This paper presents an unsupervised approach towards automatically extracting video-based guidance on object usage, from egocentric video and wearable gaze tracking, collected from multiple users while performing tasks. The approach i) discovers task relevant objects, ii) builds a model for each, iii) distinguishes different ways in which each discovered object has been used and iv) discovers the dependencies between object interactions. The work investigates using appearance, position, motion and attention, and presents results using each and a combination of relevant features. Moreover, an online scalable approach is presented and is compared to offline results. The paper proposes a method for selecting a suitable video guide to be displayed to a novice user indicating how to use an object, purely triggered by the user's gaze. The potential assistive mode can also recommend an object to be used next based on the learnt sequence of object interactions. The approach was tested on a variety of daily tasks such as initialising a printer, preparing a coffee and setting up a gym machine.