# What I See Is What You See: Joint Attention Learning for First and Third Person Video Co-analysis

arXiv: https://arxiv.org/abs/1904.07424 · HTML: https://mlsanity.com/paper/1904.07424/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2019.

## Built on by (most cited first, 17 of 17)

- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment](https://mlsanity.com/paper/2306.05526/) (2023)
- [EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding](https://mlsanity.com/paper/2406.08877/) (2024)
- [Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos](https://mlsanity.com/paper/2403.06351/) (2024)
- [Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition](https://mlsanity.com/paper/2308.11489/) (2023)
- [Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairs](https://mlsanity.com/paper/2312.02638/) (2023)
- [EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding](https://mlsanity.com/paper/2309.02423/) (2023)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)
- [O-MaMa: Learning Object Mask Matching between Egocentric and Exocentric Views](https://mlsanity.com/paper/2506.06026/) (2025)
- [Egocentric and Exocentric Methods: A Short Survey](https://mlsanity.com/paper/2410.20621/) (2024)
- [Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations](https://mlsanity.com/paper/2503.19706/) (2025)
- [From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision](https://mlsanity.com/paper/2501.05711/) (2025)
- [Multi-view Gaze Target Estimation](https://mlsanity.com/paper/2508.05857/) (2025)
- [DOMR: Establishing Cross-View Segmentation via Dense Object Matching](https://mlsanity.com/paper/2508.04050/) (2025)
- [Towards Generalizing Temporal Action Segmentation to Unseen Views](https://mlsanity.com/paper/2504.02512/) (2025)
- [ViewBridge: Curriculum Knowledge Distillation for Activity View-Invariance Under Extreme Viewpoint Changes](https://mlsanity.com/paper/2504.05451/) (2025)

## Abstract (arXiv)

In recent years, more and more videos are captured from the first-person viewpoint by wearable cameras. Such first-person video provides additional information besides the traditional third-person video, and thus has a wide range of applications. However, techniques for analyzing the first-person video can be fundamentally different from those for the third-person video, and it is even more difficult to explore the shared information from both viewpoints. In this paper, we propose a novel method for first- and third-person video co-analysis. At the core of our method is the notion of "joint attention", indicating the learnable representation that corresponds to the shared attention regions in different viewpoints and thus links the two viewpoints. To this end, we develop a multi-branch deep network with a triplet loss to extract the joint attention from the first- and third-person videos via self-supervised learning. We evaluate our method on the public dataset with cross-viewpoint video matching tasks. Our method outperforms the state-of-the-art both qualitatively and quantitatively. We also demonstrate how the learned joint attention can benefit various applications through a set of additional experiments.