# Glove2Hand: Synthesizing Natural Hand-Object Interaction from Multi-Modal Sensing Gloves

arXiv: https://arxiv.org/abs/2603.20850 · HTML: https://mlsanity.com/paper/2603.20850/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 82 references that are in the map.
- First posted on arXiv 21 Mar 2026.

## Builds on (most cited first, 30 of 60)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [gsplat: An Open-Source Library for Gaussian Splatting](https://mlsanity.com/paper/2409.06765/) (2024)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video](https://mlsanity.com/paper/2505.11709/) (2025)
- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion](https://mlsanity.com/paper/2308.16905/) (2023)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [Affordance Diffusion: Synthesizing Hand-Object Interactions](https://mlsanity.com/paper/2303.12538/) (2023)
- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control](https://mlsanity.com/paper/2503.05639/) (2025)
- [VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet](https://mlsanity.com/paper/2307.14073/) (2023)
- [DiffuEraser: A Diffusion Model for Video Inpainting](https://mlsanity.com/paper/2501.10018/) (2025)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)

## Built on by (most cited first, 3 of 3)

- [HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos](https://mlsanity.com/paper/2605.24934/) (2026)
- [AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking](https://mlsanity.com/paper/2605.21714/) (2026)
- [EgoTac: In-the-wild Tactile Prediction from Egocentric Vision](https://mlsanity.com/paper/2608.15060/) (2026)

## Abstract (arXiv)

Understanding hand-object interaction (HOI) is fundamental to computer vision, robotics, and AR/VR. However, conventional hand videos often lack essential physical information such as contact forces and motion signals, and are prone to frequent occlusions. To address the challenges, we present Glove2Hand, a framework that translates multi-modal sensing glove HOI videos into photorealistic bare hands, while faithfully preserving the underlying physical interaction dynamics. We introduce a novel 3D Gaussian hand model that ensures temporal rendering consistency. The rendered hand is seamlessly integrated into the scene using a diffusion-based hand restorer, which effectively handles complex hand-object interactions and non-rigid deformations. Leveraging Glove2Hand, we create HandSense, the first multi-modal HOI dataset featuring glove-to-hand videos with synchronized tactile and IMU signals. We demonstrate that HandSense significantly enhances downstream bare-hand applications, including video-based contact estimation and hand tracking under severe occlusion.