mlsanityData to 7 Oct 2026

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

arXiv 2106.05266 ↗First posted 9 Jun 2021cs.CVcited by 140

Not on the map. Why?

Abstract (arXiv)

Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations are scarce as even humans cannot directly label the ground-truths from a single image perfectly. To tackle these challenges, we propose a unified framework for estimating the 3D hand and object poses with semi-supervised learning. We build a joint learning framework where we perform explicit contextual reasoning between hand and object representations by a Transformer. Going beyond limited 3D annotations in a single image, we leverage the spatial-temporal consistency in large-scale hand-object videos as a constraint for generating pseudo labels in semi-supervised learning. Our method not only improves hand pose estimation in challenging real-world dataset, but also substantially improve the object pose which has fewer ground-truths per instance. By training with large-scale diverse videos, our model also generalizes better across multiple out-of-domain datasets. Project page and code: https://stevenlsw.github.io/Semi-Hand-Object

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 65 of its 90 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

65 of 90 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2017Mask R-CNNcited by 7,259
  6. 2016Layer Normalizationcited by 5,579
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
38 more
  1. 2018Image Transformercited by 770
  2. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  5. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  6. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  7. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  8. 2018PVNet: Pixel-wise Voting Network for 6DoF Pose Estimationcited by 385
  9. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  10. 2017A simple neural network module for relational reasoningcited by 345
  11. 2015LSTM: A Search Space Odysseycited by 315
  12. 2017Relation Networks for Object Detectioncited by 305
  13. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  14. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  15. 2017SSD-6D: Making RGB-based 3D detection and 6D pose estimation great againcited by 243
  16. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  17. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  18. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  19. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  20. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  21. 2018Videos as Space-Time Region Graphscited by 221
  22. 2015Unifying distillation and privileged informationcited by 218
  23. 2017Real-Time Seamless Single Shot 6D Object Pose Predictioncited by 210
  24. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  25. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  26. 2019Category-Level Articulated Object Pose Estimationcited by 158
  27. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  28. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  29. 2019H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactionscited by 139
  30. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  31. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  32. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  33. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  34. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  35. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  36. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  37. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  38. 2018Segmentation-driven 6D Object Pose Estimationcited by 104

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 20; 2023: 35; 2024: 40; 2025: 22; 2026: 12.

140 papers.

Sort:

By area:

  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  2. 2023Reconstructing Hands in 3D with Transformerscited by 298
  3. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  4. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  5. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  6. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  7. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
  8. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  9. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  10. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 95
  11. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  12. 2023An Outlook into the Future of Egocentric Visioncited by 84
38 more
  1. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  2. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  3. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  4. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  5. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  6. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  7. 2022LISA: Learning Implicit Shape and Appearance of Handscited by 63
  8. 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 62
  9. 2024GraspXL: Generating Grasping Motions for Diverse Objects at Scalecited by 59
  10. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  11. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  12. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  13. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 54
  14. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  15. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  16. 2023ContactGen: Generative Contact Modeling for Grasp Generationcited by 52
  17. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  18. 2021A Skeleton-Driven Neural Occupancy Representation for Articulated Handscited by 50
  19. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  20. 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 47
  21. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  22. 2023Detecting Human-Object Contact in Imagescited by 41
  23. 2024Deep Learning-Based Object Pose Estimation: A Comprehensive Surveycited by 39
  24. 2021Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactionscited by 37
  25. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Datacited by 37
  26. 2022CASA: Category-agnostic Skeletal Animal Reconstructioncited by 35
  27. 2024Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objectscited by 35
  28. 2023Learning Human-to-Robot Handovers from Point Cloudscited by 34
  29. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 33
  30. 2023A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Imagecited by 32
  31. 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 32
  32. 2022Consistent 3D Hand Reconstruction in Video via self-supervised Learningcited by 28
  33. 2022Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motioncited by 27
  34. 2022Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Graspscited by 27
  35. 2024A Simple Baseline for Efficient Hand Mesh Reconstructioncited by 27
  36. 2024Real3D: Scaling Up Large Reconstruction Models with Real-World Imagescited by 24
  37. 2023HMP: Hand Motion Priors for Pose and Shape Estimation from Videocited by 23
  38. 2022Interacting Hand-Object Pose Estimation via Dense Mutual Attentioncited by 22

50 most cited of 140 All 140 (JSON)