mlsanityData to 7 Oct 2026

TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding

arXiv 2401.08399 ↗First posted 16 Jan 2024cs.CVcited by 85

Not on the map. Why?

Abstract (arXiv)

Humans commonly work with multiple objects in daily life and can intuitively transfer manipulation skills to novel objects by understanding object functional regularities. However, existing technical approaches for analyzing and synthesizing hand-object manipulation are mostly limited to handling a single hand and object due to the lack of data support. To address this, we construct TACO, an extensive bimanual hand-object-interaction dataset spanning a large variety of tool-action-object compositions for daily human activities. TACO contains 2.5K motion sequences paired with third-person and egocentric views, precise hand-object 3D meshes, and action labels. To rapidly expand the data scale, we present a fully automatic data acquisition pipeline combining multi-view sensing with an optical motion capture system. With the vast research fields provided by TACO, we benchmark three generalizable hand-object-interaction tasks: compositional action recognition, generalizable hand-object motion forecasting, and cooperative grasp synthesis. Extensive experiments reveal new insights, challenges, and opportunities for advancing the studies of generalizable hand-object motion analysis and synthesis. Our data and code are available at https://taco2024.github.io.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 71 of its 88 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

71 of 88 references.

References · 71

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2023Segment Anythingcited by 8,158
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2018YOLOv3: An Incremental Improvementcited by 2,652
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  6. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  7. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  9. 2022Human Motion Diffusion Modelcited by 1,062
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  12. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
38 more
  1. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  2. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  3. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  4. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  5. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  6. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  7. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  8. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  9. 2023Track Anything: Segment Anything Meets Videoscited by 230
  10. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  11. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  12. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  13. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  14. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  15. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  16. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  17. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  18. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  19. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  20. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 112
  21. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  22. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  23. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 95
  24. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  25. 2019Context-aware Human Motion Predictioncited by 83
  26. 2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
  27. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  28. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  29. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  30. 2021Object-Region Video Transformerscited by 73
  31. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 69
  32. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  33. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 54
  34. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  35. 2021TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wildcited by 52
  36. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 52
  37. 2021A Skeleton-Driven Neural Occupancy Representation for Articulated Handscited by 50
  38. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 47

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 31; 2026: 41.

85 papers.

Sort:

By area:

  1. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  2. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  3. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  4. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  5. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  6. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  7. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  8. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  9. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 34
  10. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  11. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  12. 2024SemGrasp: Semantic Grasp Generation via Language Aligned Discretizationcited by 29
38 more
  1. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  2. 2024ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusioncited by 21
  3. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  4. 2024EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Visioncited by 18
  5. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  6. 2025OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Modelcited by 17
  7. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  8. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  9. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 14
  10. 2025DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human Referencescited by 13
  11. 2024Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrationscited by 12
  12. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 12
  13. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 11
  14. 2024QuasiSim: Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfercited by 10
  15. 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videoscited by 10
  16. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 10
  17. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
  18. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  19. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  20. 2025Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Explorationcited by 9
  21. 2025WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generationcited by 9
  22. 2025DyTact: Capturing Dynamic Contacts in Hand-Object Manipulationcited by 8
  23. 2025EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observationscited by 8
  24. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  25. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  26. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  27. 20253D Human Interaction Generation: A Surveycited by 6
  28. 2026Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstrationcited by 6
  29. 2024Gaze-guided Hand-Object Interaction Synthesis: Dataset and Methodcited by 5
  30. 2025Diffusion-based 3D Hand Motion Recovery with Intuitive Physicscited by 5
  31. 2025Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representationcited by 5
  32. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  33. 2026PAD-Hand: Physics-Aware Diffusion for Hand Motion Recoverycited by 4
  34. 2026SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wildcited by 4
  35. 2025HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordinationcited by 3
  36. 2025BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assemblycited by 3
  37. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  38. 2026From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Datacited by 3

50 most cited of 85 All 85 (JSON)