mlsanityData to 7 Oct 2026

Actor and Observer: Joint Modeling of First and Third-Person Videos

arXiv 1804.09627 ↗First posted 25 Apr 2018cs.CVcited by 102

Not on the map. Why?

Abstract (arXiv)

Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor). Despite this, learning such models for human action recognition has not been achievable due to the lack of data. This paper takes a step in this direction, with the introduction of Charades-Ego, a large-scale dataset of paired first-person and third-person videos, involving 112 people, with 4000 paired videos. This enables learning the link between the two, actor and observer perspectives. Thereby, we address one of the biggest bottlenecks facing egocentric vision research, providing a link from first-person to the abundant third-person data on the web. We use this data to learn a joint representation of first and third-person videos, with only weak supervision, and show its effectiveness for transferring knowledge from the third-person to the first-person domain.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 18; 2023: 21; 2024: 23; 2025: 17; 2026: 9.

102 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  3. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  4. 2022Egocentric Video-Language Pretrainingcited by 231
  5. 2022Learning Video Representations from Large Language Modelscited by 204
  6. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  7. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  8. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  9. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  10. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  11. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  12. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
38 more
  1. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  2. 2023An Outlook into the Future of Egocentric Visioncited by 84
  3. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  4. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  5. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  6. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  7. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
  8. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  9. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  10. 2021Learning Cross-modal Contrastive Features for Video Domain Adaptationcited by 47
  11. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  12. 2021Representation Learning via Global Temporal Alignment and Cycle-Consistencycited by 43
  13. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  14. 2021Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outlierscited by 39
  15. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  16. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  17. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  18. 2021Self-Regulated Learning for Egocentric Video Activity Anticipationcited by 32
  19. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  20. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  21. 2021Learning Visual Affordance Grounding from Demonstration Videoscited by 25
  22. 2022Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessmentcited by 25
  23. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  24. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  25. 2022Grounded Affordance from Exocentric Viewcited by 21
  26. 2023Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognitioncited by 20
  27. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  28. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  29. 2023Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairscited by 17
  30. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  31. 2023Cross-view Action Recognition Understanding From Exocentric to Egocentric Perspectivecited by 14
  32. 2023EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understandingcited by 14
  33. 2022Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Spacecited by 13
  34. 2024Intention-driven Ego-to-Exo Video Generationcited by 13
  35. 2024BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulationcited by 13
  36. 2022Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videoscited by 12
  37. 2023Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videoscited by 11
  38. 2024Step Differences in Instructional Videocited by 11

50 most cited of 102 All 102 (JSON)