mlsanityData to 7 Oct 2026

The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines

arXiv 2005.00343 ↗First posted 29 Apr 2020cs.CVcited by 221

Not on the map. Why?

Abstract (arXiv)

Since its introduction in 2018, EPIC-KITCHENS has attracted attention as the largest egocentric video benchmark, offering a unique viewpoint on people's interaction with objects, their attention, and even intention. In this paper, we detail how this large-scale dataset was captured by 32 participants in their native kitchen environments, and densely annotated with actions and object interactions. Our videos depict nonscripted daily activities, as recording is started every time a participant entered their kitchen. Recording took place in 4 countries by participants belonging to 10 different nationalities, resulting in highly diverse kitchen habits and cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.2K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos after recording, thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and. anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. We introduce new baselines that highlight the multimodal nature of the dataset and the importance of explicit temporal modelling to discriminate fine-grained actions e.g. 'closing a tap' from 'opening' it up.

Key facts. Cited by 221 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 21; 2023: 34; 2024: 42; 2025: 56; 2026: 50.

221 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  3. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  4. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  5. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  8. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  9. 2021Learning Open-World Object Proposals without Learning to Classifycited by 109
  10. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  11. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  12. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
38 more
  1. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 79
  2. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  3. 2023OtterHD: A High-Resolution Multi-modality Modelcited by 72
  4. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  5. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  6. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  7. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  8. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  9. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  10. 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 51
  11. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 49
  12. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  13. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  14. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  15. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  16. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  17. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  18. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  19. 2026Infinite Worlds with Versatile Interactionscited by 37
  20. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  21. 2023MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognitioncited by 36
  22. 2022Intention-Conditioned Long-Term Human Egocentric Action Forecastingcited by 35
  23. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  24. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  25. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  26. 2022Egocentric Prediction of Action Target in 3Dcited by 31
  27. 2024OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understandingcited by 31
  28. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  29. 2023PALM: Predicting Actions through Language Modelscited by 28
  30. 2024EgoLM: Multi-Modal Language Model of Egocentric Motionscited by 28
  31. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  32. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  33. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  34. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  35. 2025RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulationcited by 24
  36. 2021Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videoscited by 23
  37. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  38. 2024From an Image to a Scene: Learning to Imagine the World from a Million 360 Videoscited by 23

50 most cited of 221 All 221 (JSON)