mlsanityData to 7 Oct 2026

EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World

arXiv 2403.16182 ↗First posted 24 Mar 2024cs.CVcited by 93

Not on the map. Why?

Abstract (arXiv)

Being able to map the activities of others into one's own point of view is one fundamental human skill even from a very early age. Taking a step toward understanding this human ability, we introduce EgoExoLearn, a large-scale dataset that emulates the human demonstration following process, in which individuals record egocentric videos as they execute tasks guided by demonstration videos. Focusing on the potential applications in daily assistance and professional support, EgoExoLearn contains egocentric and demonstration video data spanning 120 hours captured in daily life scenarios and specialized laboratories. Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints. To this end, we present benchmarks such as cross-view association, cross-view action planning, and cross-view referenced skill assessment, along with detailed analysis. We expect EgoExoLearn can serve as an important resource for bridging the actions across views, thus paving the way for creating AI agents capable of seamlessly learning by observing humans in the real world. Code and data can be found at: https://github.com/OpenGVLab/EgoExoLearn

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 109 of its 172 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

109 of 172 references.

References · 109

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
38 more
  1. 2021Multiscale Vision Transformerscited by 896
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2020Rescaling Egocentric Visioncited by 529
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  11. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  12. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  13. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  14. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  15. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  16. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  17. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  18. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  19. 2022Egocentric Video-Language Pretrainingcited by 231
  20. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  21. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  22. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  23. 2022Learning Video Representations from Large Language Modelscited by 204
  24. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 201
  25. 2021Anticipative Video Transformercited by 197
  26. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  27. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  28. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  29. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  30. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  31. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  32. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  33. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  34. 2018Disentangled Sequential Autoencodercited by 117
  35. 2020Multi-Modal Domain Adaptation for Fine-Grained Action Recognitioncited by 115
  36. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  37. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  38. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110

50 most cited of 109 All 109 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 48; 2026: 25.

93 papers.

Sort:

By area:

  1. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  2. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  3. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  4. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  5. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
  6. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  7. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 26
  8. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  9. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QAcited by 24
  10. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  11. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  12. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
38 more
  1. 2024A Comprehensive Survey of Action Quality Assessment: Method and Benchmarkcited by 20
  2. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  3. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  4. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  5. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  6. 2024Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognitioncited by 18
  7. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 18
  8. 2025BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimationcited by 17
  9. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16
  10. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  11. 2025Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesiscited by 12
  12. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 11
  13. 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspectivecited by 10
  14. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understandingcited by 10
  15. 2025UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speakingcited by 10
  16. 2025TeleEgo: Benchmarking Egocentric AI Assistants in the Wildcited by 9
  17. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  18. 2025ExAct: A Video-Language Benchmark for Expert Action Analysiscited by 8
  19. 2025Towards Interactive Intelligence for Digital Humanscited by 8
  20. 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 8
  21. 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
  22. 2024CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentationcited by 7
  23. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  24. 2024Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuningcited by 7
  25. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 7
  26. 2025A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directionscited by 7
  27. 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistantcited by 7
  28. 2024Egocentric and Exocentric Methods: A Short Surveycited by 6
  29. 2024Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videoscited by 6
  30. 2024TechCoach: Towards Technical-Point-Aware Descriptive Action Coachingcited by 6
  31. 2025Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representationscited by 6
  32. 2025SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimationcited by 6
  33. 2025PATS: Proficiency-Aware Temporal Sampling for Multi-View Sports Skill Assessmentcited by 5
  34. 2025ProMQA-Assembly: Multimodal Procedural QA Dataset on Assemblycited by 5
  35. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  36. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervisioncited by 4
  37. 2025ProfVLM: A lightweight video-language model for multi-view proficiency estimationcited by 4
  38. 2025EgoExo-Con: Exploring View-Invariant Video Temporal Understandingcited by 4

50 most cited of 93 All 93 (JSON)