V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction
Not on the map. Why?
Abstract (arXiv)
In whiteboard-based remote communication, the seamless integration of drawn content and hand-screen interactions is essential for an immersive user experience. Previous methods either require bulky device setups for capturing hand gestures or fail to accurately track the hand poses from capacitive images. In this paper, we present a real-time method for precise tracking 3D poses of both hands from capacitive video frames. To this end, we develop a deep neural network to identify hands and infer hand joint positions from capacitive frames, and then recover 3D hand poses from the hand-joint positions via a constrained inverse kinematic solver. Additionally, we design a device setup for capturing high-quality hand-screen interaction data and obtained a more accurate synchronized capacitive video and hand pose dataset. Our method improves the accuracy and stability of 3D hand tracking for capacitive frames while maintaining a compact device setup for remote communication. We validate our scheme design and its superior performance on 3D hand pose tracking and demonstrate the effectiveness of our method in whiteboard-based remote communication. Our code, model, and dataset are available at https://V-Hands.github.io.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 91 references that are in the map. First posted on arXiv 20 Sep 2024.
Builds on
28 of 91 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2017Squeeze-and-Excitation Networkscited by 4,763
- 2017Integral Human Pose Regressioncited by 255
- 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
- 20193D Hand Shape and Pose from Images in the Wildcited by 190
- 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
- 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
- 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
- 2019Learnable Triangulation of Human Posecited by 160
- 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
16 more
- 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
- 2021Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Cameracited by 70
- 2020Rethinking the Heatmap Regression for Bottom-up Human Pose Estimationcited by 69
- 2019A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Imagecited by 56
- 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36
- 2022Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videoscited by 35
- 2020SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimationcited by 34
- 2020Two-hand Global 3D Pose Estimation Using Monocular RGBcited by 27
- 2022Interacting Hand-Object Pose Estimation via Dense Mutual Attentioncited by 22
- 2017Large-scale Multiview 3D Hand Pose Datasetcited by 20
- 2020JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Imagecited by 20
- 2023A step towards understanding why classification helps regressioncited by 15
- 2022Heatmap Distribution Matching for Human Pose Estimationcited by 9
- 2020Learning to Compress Videos without Computing Motioncited by 7
- 2021PeCLR: Self-Supervised 3D Hand Pose Estimation from monocular RGB via Equivariant Contrastive Learningcited by 2
- 2023RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touchcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.