mlsanityData to 7 Oct 2026

HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructions

arXiv 2308.01477 ↗First posted 2 Aug 2023cs.RO cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

We present the HANDAL dataset for category-level object pose estimation and affordance prediction. Unlike previous datasets, ours is focused on robotics-ready manipulable objects that are of the proper size and shape for functional grasping by robot manipulators, such as pliers, utensils, and screwdrivers. Our annotation process is streamlined, requiring only a single off-the-shelf camera and semi-automated processing, allowing us to produce high-quality 3D annotations without crowd-sourcing. The dataset consists of 308k annotated image frames from 2.2k videos of 212 real-world objects in 17 categories. We focus on hardware and kitchen tool objects to facilitate research in practical scenarios in which a robot manipulator needs to interact with the environment beyond simple pushing or indiscriminate grasping. We outline the usefulness of our dataset for 6-DoF category-level pose+scale estimation and related tasks. We also provide 3D reconstructed meshes of all objects, and we outline some of the bottlenecks to be addressed for democratizing the collection of datasets like this one.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

34 of 44 references.

References · 34

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2017Mask R-CNNcited by 7,259
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  6. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  7. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  8. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  9. 2018BOP: Benchmark for 6D Object Pose Estimationcited by 254
  10. 2018The Perfect Match: 3D Point Cloud Matching with Smoothed Densitiescited by 169
  11. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  12. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
22 more
  1. 2021FS-Net: Fast Shape-based Network for Category-Level 6D Object Pose Estimation with Decoupled Rotation Mechanismcited by 121
  2. 2021DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistencycited by 103
  3. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 87
  4. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  5. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 85
  6. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  7. 2022ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimizationcited by 70
  8. 2022CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimationcited by 69
  9. 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
  10. 2021CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulationcited by 49
  11. 2022PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objectscited by 45
  12. 2021Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Imagecited by 42
  13. 2022SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimationcited by 41
  14. 2023TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimationcited by 32
  15. 2020Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Handscited by 26
  16. 2021iCaps: Iterative Category-level Object Pose and Shape Estimationcited by 23
  17. 2022Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimationcited by 19
  18. 2020MobilePose: Real-Time Pose Estimation for Unseen Objects with Weak Shape Supervisioncited by 16
  19. 2022RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Controlcited by 10
  20. 2021On Minimizing the Number of Running Buffers for Tabletop Rearrangementcited by 5
  21. 2022Toward Efficient Task Planning for Dual-Arm Tabletop Object Rearrangementcited by 5
  22. 2022Parallel Monte Carlo Tree Search with Batched Rigid-body Simulations for Speeding up Long-Horizon Episodic Robot Planningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 19; 2026: 18.

47 papers.

Sort:

By area:

  1. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  2. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 46
  3. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 26
  4. 2024HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interactioncited by 25
  5. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 19
  6. 2025BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimationcited by 18
  7. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  8. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 12
  9. 2023PACE: A Large-Scale Dataset with Pose Annotations in Cluttered Environmentscited by 11
  10. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  11. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
  12. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
35 more
  1. 2025Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priorscited by 9
  2. 2024UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Modelscited by 6
  3. 2025AffordanceSAM: Segment Anything Once More in Affordance Groundingcited by 6
  4. 2024Segmenting Object Affordances: Reproducibility and Sensitivity to Scalecited by 3
  5. 2024GFreeDet: Exploiting Gaussian Splatting and Foundation Models for Model-free Unseen Object Detection in the BOP Challenge 2024cited by 3
  6. 2025MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentationcited by 3
  7. 2023NeuralLabeling: A versatile toolset for labeling vision datasets using Neural Radiance Fieldscited by 2
  8. 2025Weakly-Supervised Learning of Dense Functional Correspondencescited by 2
  9. 2025Beyond 'Templates': Category-Agnostic Object Pose, Size, and Shape Estimation from a Single Viewcited by 2
  10. 2026Event6D: Event-based Novel Object 6D Pose Trackingcited by 2
  11. 2024ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?cited by 1
  12. 2025Vision6D: 3D-to-2D Interactive Visualization and Annotation Tool for 6D Pose Estimationcited by 1
  13. 2025Visual Affordance Prediction: Survey and Reproducibilitycited by 1
  14. 2025MR6D: Benchmarking 6D Pose Estimation for Mobile Robotscited by 1
  15. 2026AffordanceGrasp-R1:Leveraging Reasoning-Based Affordance Segmentation with Reinforcement Learning for Robotic Graspingcited by 1
  16. 2026IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and toolscited by 1
  17. 2026Panoramic Affordance Predictioncited by 1
  18. 2026Towards Symmetry-sensitive Pose Estimation: A Rotation Representation for Symmetric Object Classescited by 1
  19. 2026SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scalecited by 1
  20. 2026SpatialCLI: Learning to Reason With Spatial Tools, Then Without Themcited by 1
  21. 2025DynOPETs: A Versatile Benchmark for Dynamic Object Pose Estimation and Tracking in Moving Camera Scenarioscited by 0
  22. 2025Computer Vision for Objects used in Group Work: Challenges and Opportunitiescited by 0
  23. 2025MVTOP: Multi-View Transformer-based Object Pose-Estimationcited by 0
  24. 2025NeuralMeshing: Complete Object Mesh Extraction from Casual Capturescited by 0
  25. 2025OPFormer: Object Pose Estimation leveraging foundation model with geometric encodingcited by 0
  26. 2026Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perceptioncited by 0
  27. 2026PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Modelscited by 0
  28. 2026Exploring 6D Object Pose Estimation with Deformationcited by 0
  29. 2026Every9D-21M: Large-Scale Real-World 9D Canonicalization of Everyday Objectscited by 0
  30. 2026UniPose9D: Universal Category-Agnostic Object Pose Estimationcited by 0
  31. 2026VLAff: Vision-Language-Affordance Model for Unified Actionable Affordancescited by 0
  32. 2026Reproducible Multimodal Affordance Predictioncited by 0
  33. 2026PhysBrain 1.5: From Vision-Language Models to Physical Foundation Modelscited by 0
  34. 2026UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perceptioncited by 0
  35. 2026GATOR: Generative and Agentic 3D Object Reconstruction From Casual Imagescited by 0