mlsanityData to 7 Oct 2026

Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos

arXiv 2104.07905 ↗First posted 16 Apr 2021cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 56 of its 76 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

56 of 76 references.

References · 56

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  9. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  10. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  12. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
38 more
  1. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  2. 2017Temporal Relational Reasoning in Videoscited by 324
  3. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  5. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  6. 2015Unifying distillation and privileged informationcited by 218
  7. 2015Cross Modal Distillation for Supervision Transfercited by 173
  8. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  9. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  10. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  11. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  12. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  13. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  14. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  15. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  16. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  17. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  18. 2019Bridging the Domain Gap for Ground-to-Aerial Image Matchingcited by 95
  19. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  20. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  21. 2018Cross-View Image Synthesis using Conditional GANscited by 69
  22. 2018Object Level Visual Reasoning in Videoscited by 67
  23. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  24. 2018LSTA: Long Short-Term Attention for Egocentric Action Recognitioncited by 57
  25. 2019Next-Active-Object prediction from Egocentric Videoscited by 55
  26. 2016Going Deeper into First-Person Activity Recognitioncited by 48
  27. 2016Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Videocited by 45
  28. 2016First-Person Activity Forecasting with Online Inverse Reinforcement Learningcited by 40
  29. 2017Graph Distillation for Action Detection with Privileged Modalitiescited by 34
  30. 2015Predicting Important Objects for Egocentric Video Summarizationcited by 21
  31. 2019DistInit: Learning Video Representations Without a Single Labeled Videocited by 21
  32. 2017Identifying First-person Camera Wearers in Third-person Videoscited by 18
  33. 2018Joint Person Segmentation and Identification in Synchronized First- and Third-person Videoscited by 18
  34. 2016Ego2Top: Matching Viewers in Egocentric and Top-view Videoscited by 17
  35. 2019What I See Is What You See: Joint Attention Learning for First and Third Person Video Co-analysiscited by 17
  36. 2020Exocentric to Egocentric Image Generation via Parallel Generative Adversarial Networkcited by 17
  37. 2020Egocentric Object Manipulation Graphscited by 17
  38. 2017Summarizing First-Person Videos from Third Persons' Points of Viewscited by 15

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 12; 2023: 21; 2024: 20; 2025: 30; 2026: 11.

98 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  4. 2022Egocentric Video-Language Pretrainingcited by 231
  5. 2022Learning Video Representations from Large Language Modelscited by 204
  6. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  7. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  8. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  9. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  10. 2023An Outlook into the Future of Egocentric Visioncited by 84
  11. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  12. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
38 more
  1. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  2. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  3. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  4. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  5. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  6. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  7. 2023Egocentric Audio-Visual Object Localizationcited by 40
  8. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  9. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  10. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  11. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  12. 2022Egocentric Prediction of Action Target in 3Dcited by 31
  13. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  14. 2023EgoTracks: A Long-term Egocentric Visual Object Tracking Datasetcited by 27
  15. 2023Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learningcited by 26
  16. 2021Learning Visual Affordance Grounding from Demonstration Videoscited by 25
  17. 2022EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queriescited by 25
  18. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  19. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 25
  20. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QAcited by 24
  21. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  22. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  23. 2022Grounded Affordance from Exocentric Viewcited by 21
  24. 2023Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognitioncited by 20
  25. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  26. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  27. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  28. 2023Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairscited by 17
  29. 2025HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstructioncited by 15
  30. 2023Cross-view Action Recognition Understanding From Exocentric to Egocentric Perspectivecited by 14
  31. 2023EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understandingcited by 14
  32. 2024Intention-driven Ego-to-Exo Video Generationcited by 13
  33. 2023Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videoscited by 11
  34. 2024EgoPet: Egomotion and Interaction Data from an Animal's Perspectivecited by 11
  35. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  36. 2023Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?cited by 10
  37. 2024POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View Worldcited by 10
  38. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9

50 most cited of 98 All 98 (JSON)