# Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos

arXiv: https://arxiv.org/abs/2104.07905 · HTML: https://mlsanity.com/paper/2104.07905/

## Key facts

- Cited by 99 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 76 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 30 of 56)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions](https://mlsanity.com/paper/1904.05349/) (2019)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [EGO-TOPO: Environment Affordances from Egocentric Video](https://mlsanity.com/paper/2001.04583/) (2020)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention](https://mlsanity.com/paper/1905.09035/) (2019)
- [Actor and Observer: Joint Modeling of First and Third-Person Videos](https://mlsanity.com/paper/1804.09627/) (2018)
- [Bridging the Domain Gap for Ground-to-Aerial Image Matching](https://mlsanity.com/paper/1904.11045/) (2019)

## Built on by (most cited first, 30 of 99)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World](https://mlsanity.com/paper/2403.16182/) (2024)
- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Learning Affordance Grounding from Exocentric Images](https://mlsanity.com/paper/2203.09905/) (2022)
- [HierVL: Learning Hierarchical Video-Language Embeddings](https://mlsanity.com/paper/2301.02311/) (2023)
- [LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding](https://mlsanity.com/paper/2303.09665/) (2023)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation](https://mlsanity.com/paper/2408.16730/) (2024)
- [Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment](https://mlsanity.com/paper/2306.05526/) (2023)
- [Retrieval-Augmented Egocentric Video Captioning](https://mlsanity.com/paper/2401.00789/) (2024)
- [With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition](https://mlsanity.com/paper/2111.01024/) (2021)
- [EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation](https://mlsanity.com/paper/2411.08380/) (2024)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Audio-Adaptive Activity Recognition Across Video Domains](https://mlsanity.com/paper/2203.14240/) (2022)
- [Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting](https://mlsanity.com/paper/2307.08243/) (2023)
- [Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videos](https://mlsanity.com/paper/2403.06351/) (2024)
- [Egocentric Prediction of Action Target in 3D](https://mlsanity.com/paper/2203.13116/) (2022)
- [Ego-Only: Egocentric Action Detection without Exocentric Transferring](https://mlsanity.com/paper/2301.01380/) (2023)
- [Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video](https://mlsanity.com/paper/2510.14560/) (2025)
- [EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset](https://mlsanity.com/paper/2301.03213/) (2023)
- [Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning](https://mlsanity.com/paper/2301.10931/) (2023)
- [Learning Visual Affordance Grounding from Demonstration Videos](https://mlsanity.com/paper/2108.05675/) (2021)
- [EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries](https://mlsanity.com/paper/2212.06969/) (2022)

## Abstract (arXiv)

We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.