mlsanityData to 14 Oct 2026

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives

arXiv 2311.18259 ↗First posted 30 Nov 2023cs.CV cs.AIcited by 490

Not on the map. Why?

Cited by 56 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community. Project page: http://ego-exo4d-data.org/

Key facts. Cited by 490 papers in the map (data through 14 Oct 2026). Builds on 155 of its 212 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

155 of 212 references.

References · 155

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2023Segment Anythingcited by 8,234
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
38 more
  1. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  7. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  9. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  11. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  12. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  13. 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 811
  14. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  15. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  16. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  17. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  18. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  19. 2019Learned Step Size Quantizationcited by 628
  20. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 627
  21. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  22. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  23. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  24. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  25. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  26. 2020Rescaling Egocentric Visioncited by 540
  27. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  28. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  29. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  30. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  31. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  32. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  33. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  34. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  35. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  36. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 307
  37. 2022Novel View Synthesis with Diffusion Modelscited by 300
  38. 2018Model compression via distillation and quantizationcited by 289

50 most cited of 155 All 155 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 87; 2025: 188; 2026: 213.

490 papers.

Sort:

By area:

  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  3. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  4. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  5. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  6. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  7. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  8. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  9. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  10. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 128
  11. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 117
  12. 2024Large Language Models for Time Series: A Surveycited by 114
38 more
  1. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  2. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  3. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  4. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  5. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  6. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  7. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 88
  8. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  9. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  10. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 82
  11. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  12. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 78
  13. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  14. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  15. 2026SAM 3D Body: Robust Full-Body Human Mesh Recoverycited by 69
  16. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  17. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 68
  18. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  19. 2025Towards Understanding Camera Motions in Any Videocited by 64
  20. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 62
  21. 2025Embodied AI Agents: Modeling the Worldcited by 61
  22. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 58
  23. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  24. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 57
  25. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
  26. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  27. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  28. 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 51
  29. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 50
  30. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  31. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 50
  32. 20254DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videoscited by 49
  33. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 48
  34. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 47
  35. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  36. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 45
  37. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  38. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 41

50 most cited of 490 All 490 (JSON)