mlsanityData to 7 Oct 2026

Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities

arXiv 2203.14712 ↗First posted 28 Mar 2022cs.CVcited by 284

Not on the map. Why?

Abstract (arXiv)

Assembly101 is a new procedural activity dataset featuring 4321 videos of people assembling and disassembling 101 "take-apart" toy vehicles. Participants work without fixed instructions, and the sequences feature rich and natural variations in action ordering, mistakes, and corrections. Assembly101 is the first multi-view action dataset, with simultaneous static (8) and egocentric (4) recordings. Sequences are annotated with more than 100K coarse and 1M fine-grained action segments, and 18M 3D hand poses. We benchmark on three action understanding tasks: recognition, anticipation and temporal segmentation. Additionally, we propose a novel task of detecting mistakes. The unique recording format and rich set of annotations allow us to investigate generalization to new toys, cross-view transfer, long-tailed distributions, and pose vs. appearance. We envision that Assembly101 will serve as a new challenge to investigate various activity understanding problems.

Key facts. Cited by 284 papers in the map (data through 7 Oct 2026). Builds on 42 of its 69 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

42 of 69 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
30 more
  1. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  2. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  3. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  4. 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
  5. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  6. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  7. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  8. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  9. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 339
  10. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  11. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  12. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  13. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  14. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  15. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  16. 2017Third-Person Imitation Learningcited by 103
  17. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  18. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  19. 2016Learning To Score Olympic Eventscited by 87
  20. 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
  21. 2018Action Quality Assessment Across Multiple Actionscited by 78
  22. 2020Spatial Temporal Transformer Network for Skeleton-based Action Recognitioncited by 76
  23. 2018The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videoscited by 68
  24. 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
  25. 2020Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representationcited by 57
  26. 2018Zero-Shot Anticipation for Instructional Activitiescited by 43
  27. 2020Long-Term Anticipation of Activities with Cycle Consistencycited by 39
  28. 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
  29. 2021Coarse to Fine Multi-Resolution Temporal Convolutional Networkcited by 28
  30. 2017Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognitioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 42; 2024: 66; 2025: 86; 2026: 83.

284 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  3. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  4. 2023Reconstructing Hands in 3D with Transformerscited by 298
  5. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  6. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  7. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  8. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  9. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  10. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  11. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  12. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
38 more
  1. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  2. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  3. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  4. 2023An Outlook into the Future of Egocentric Visioncited by 84
  5. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 79
  6. 2023AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimationcited by 64
  7. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  8. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  9. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  10. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  11. 2023Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignmentcited by 46
  12. 2023IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Settingcited by 46
  13. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  14. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  15. 2023Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled videocited by 40
  16. 2024Aria Everyday Activities Datasetcited by 38
  17. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  18. 2023How Much Temporal Long-Term Context is Needed for Action Segmentation?cited by 36
  19. 2024PREGO: online mistake detection in PRocedural EGOcentric videoscited by 36
  20. 2023A Dataset of Relighted 3D Interacting Handscited by 35
  21. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  22. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
  23. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  24. 2024Put Myself in Your Shoes: Lifting the Egocentric Perspective from Exocentric Videoscited by 32
  25. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  26. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  27. 2024SemGrasp: Semantic Grasp Generation via Language Aligned Discretizationcited by 29
  28. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  29. 2023Learning and Verification of Task Structure in Instructional Videoscited by 26
  30. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 26
  31. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  32. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 24
  33. 2022C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentationcited by 23
  34. 2023Interactive Spatiotemporal Token Attention Network for Skeleton-based General Interactive Action Recognitioncited by 23
  35. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  36. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  37. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  38. 2024emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimationcited by 20

50 most cited of 284 All 284 (JSON)