mlsanityData to 13 Oct 2026

HD-EPIC: A Highly-Detailed Egocentric Video Dataset

arXiv 2502.04144 ↗First posted 6 Feb 2025cs.CVcited by 102

Not on the map. Why?

Abstract (arXiv)

We present a validation dataset of newly-collected kitchen-based egocentric videos, manually annotated with highly detailed and interconnected ground-truth labels covering: recipe steps, fine-grained actions, ingredients with nutritional values, moving objects, and audio annotations. Importantly, all annotations are grounded in 3D through digital twinning of the scene, fixtures, object locations, and primed with gaze. Footage is collected from unscripted recordings in diverse home environments, making HDEPIC the first dataset collected in-the-wild but with detailed annotations matching those in controlled lab environments. We show the potential of our highly-detailed annotations through a challenging VQA benchmark of 26K questions assessing the capability to recognise recipes, ingredients, nutrition, fine-grained actions, 3D perception, object motion, and gaze direction. The powerful long-context Gemini Pro only achieves 38.5% on this benchmark, showcasing its difficulty and highlighting shortcomings in current VLMs. We additionally assess action recognition, sound recognition, and long-term video-object segmentation on HD-EPIC. HD-EPIC is 41 hours of video in 9 kitchens with digital twins of 413 kitchen fixtures, capturing 69 recipes, 59K fine-grained actions, 51K audio events, 20K object movements and 37K object masks lifted to 3D. On average, we have 263 annotations per minute of our unscripted videos.

Key facts. Cited by 102 papers in the map (data through 13 Oct 2026). Builds on 68 of its 97 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

68 of 97 references.

References · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2023Segment Anythingcited by 8,195
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,096
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,820
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  10. 2018SlowFast Networks for Video Recognitioncited by 1,864
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,465
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,389
38 more
  1. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,353
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,308
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  4. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 949
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 792
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 760
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 672
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 648
  9. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 601
  11. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 583
  12. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 567
  13. 2020Rescaling Egocentric Visioncited by 532
  14. 2024Long Context Transfer from Language to Visioncited by 507
  15. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  16. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 406
  17. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 382
  18. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 323
  19. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 311
  20. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 285
  21. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 232
  22. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  23. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  24. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 205
  25. 2018Constituency Parsing with a Self-Attentive Encodercited by 197
  26. 2023Putting the Object Back into Video Object Segmentationcited by 185
  27. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  28. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 168
  29. 2019The VIA Annotation Software for Images, Audio and Videocited by 167
  30. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 167
  31. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 161
  32. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  33. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 134
  34. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 124
  35. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 103
  36. 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
  37. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 69
  38. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 32; 2026: 70.

102 papers.

Sort:

By area:

  1. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 97
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  4. 2026World Action Models: The Next Frontier in Embodied AIcited by 48
  5. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 30
  6. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videoscited by 21
  7. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoningcited by 18
  8. 2026From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agentscited by 15
  9. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  10. 2025EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understandingcited by 11
  11. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  12. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
38 more
  1. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 11
  2. 2026Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wildcited by 11
  3. 2026World Action Models: A Surveycited by 11
  4. 2025Developing Vision-Language-Action Model from Egocentric Videoscited by 10
  5. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  6. 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 8
  7. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  8. 2026FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulationcited by 7
  9. 2025Understanding Dynamic Scenes in Ego Centric 4D Point Cloudscited by 6
  10. 2025Aria Gen 2 Pilot Datasetcited by 6
  11. 2026EgoAdapt: A Multi-Scene Egocentric Adaptation Method for CVPR 2026 HD-EPIC VQA Challengecited by 6
  12. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  13. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 5
  14. 2026MolmoMotion: Forecasting Point Trajectories in 3D with Language Instructioncited by 5
  15. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 5
  16. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  17. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  18. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  19. 2025HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answeringcited by 4
  20. 2025SceneDiff: A Benchmark and Method for Multiview Object Change Detectioncited by 4
  21. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 4
  22. 2026Optimizing Multimodal LLMs for Egocentric Video Understanding: A Solution for the HD-EPIC VQA Challengecited by 3
  23. 2026Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AIcited by 3
  24. 2026neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editingcited by 3
  25. 2026SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memorycited by 3
  26. 2025Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Researchcited by 2
  27. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2
  28. 2025Unique Lives, Shared World: Learning from Single-Life Videoscited by 2
  29. 2026A Closed-Loop Multi-Agent System Driven by LLMs for Meal-Level Personalized Nutrition Managementcited by 2
  30. 2026Ego-1K -- A Large-Scale Multiview Video Dataset for Egocentric Visioncited by 2
  31. 2026Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videoscited by 2
  32. 2026EgoSelf: From Memory to Personalized Egocentric Assistantcited by 2
  33. 2026StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Videocited by 2
  34. 2026Whareformer: Learning to Track What is Where in Long Egocentric Videoscited by 2
  35. 2025Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reachcited by 1
  36. 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinkingcited by 1
  37. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
  38. 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Nextcited by 1

50 most cited of 102 All 102 (JSON)