mlsanityData to 7 Oct 2026

HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding

arXiv 2307.05721 ↗First posted 9 Jul 2023cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Understanding comprehensive assembly knowledge from videos is critical for futuristic ultra-intelligent industry. To enable technological breakthrough, we present HA-ViD - the first human assembly video dataset that features representative industrial assembly scenarios, natural procedural knowledge acquisition process, and consistent human-robot shared annotations. Specifically, HA-ViD captures diverse collaboration patterns of real-world assembly, natural human behaviors and learning progression during assembly, and granulate action annotations to subject, action verb, manipulated object, target object, and tool. We provide 3222 multi-view, multi-modality videos (each video contains one assembly task), 1.5M frames, 96K temporal labels and 2M spatial labels. We benchmark four foundational video understanding tasks: action recognition, action segmentation, object detection and multi-object tracking. Importantly, we analyze their performance for comprehending knowledge in assembly progress, process efficiency, task collaboration, skill parameters and human intention. Details of HA-ViD is available at: https://iai-hrc.github.io/ha-vid.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 32 of its 60 references that are in the map. First posted on arXiv 9 Jul 2023.

Builds on

32 of 60 references.

References · 32

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  7. 2018Datasheets for Datasetscited by 1,371
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  9. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  10. 2016Simple Online and Realtime Trackingcited by 862
  11. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  12. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
20 more
  1. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  3. 2018Object Detection with Deep Learning: A Reviewcited by 524
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  5. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  6. 2020MOT20: A benchmark for multi object tracking in crowded scenescited by 355
  7. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  8. 2014Multiple Object Tracking: A Literature Reviewcited by 165
  9. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  10. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  11. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  12. 2022PYSKL: Towards Good Practices for Skeleton Action Recognitioncited by 83
  13. 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
  14. 2022Future Transformer for Long-term Action Anticipationcited by 57
  15. 2020Temporal Relational Modeling with Self-Supervision for Action Segmentationcited by 29
  16. 2022C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentationcited by 23
  17. 2021ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videoscited by 20
  18. 2019Adversarial Skill Networks: Unsupervised Robot Skill Learning from Videocited by 17
  19. 2017Human Pose Forecasting via Deep Markov Modelscited by 16
  20. 2022Video-based Human-Object Interaction Detection from Tubelet Tokenscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 10.

18 papers.

Sort:

By area:

  1. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  2. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  3. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 18
  4. 2024IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videoscited by 11
  5. 2026MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstructioncited by 11
  6. 2026IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assemblycited by 8
  7. 2024TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Linescited by 5
  8. 2024SCANet: Correcting LEGO Assembly Errors with Self-Correct Assembly Networkcited by 3
  9. 2026OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environmentscited by 2
  10. 2026AssemblyBench: Physics-Aware Assembly of Complex Industrial Objectscited by 2
  11. 2025AssemblyHands-X: Modeling 3D Hand-Body Coordination for Understanding Bimanual Human Activitiescited by 1
  12. 2026Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioningcited by 1
6 more
  1. 2025Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challengescited by 0
  2. 2026Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentationcited by 0
  3. 2026Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videoscited by 0
  4. 2026Lag-aware cross-hand alignment for dual-hand action segmentationcited by 0
  5. 2026HEIR: Learning Human-Entity Interactions with Functional Rolescited by 0
  6. 2026Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognitioncited by 0