# RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

arXiv: https://arxiv.org/abs/2302.14365 · HTML: https://mlsanity.com/paper/2302.14365/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 89 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 10 of 10)

- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)
- [RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video](https://mlsanity.com/paper/2106.11725/) (2021)
- [VirtualCube: An Immersive 3D Video Communication System](https://mlsanity.com/paper/2112.06730/) (2021)
- [Multi-View Video-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2109.11747/) (2021)
- [A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systems](https://mlsanity.com/paper/2102.05998/) (2021)
- ["Can I Touch This?": Survey of Virtual Reality Interactions via Haptic Solutions](https://mlsanity.com/paper/2101.11278/) (2021)
- [A survey on haptic technologies for mobile augmented reality](https://mlsanity.com/paper/1709.00698/) (2017)
- [Evoking realistic affective touch experiences in virtual reality](https://mlsanity.com/paper/2202.13389/) (2022)

## Built on by (most cited first, 2 of 2)

- [V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction](https://mlsanity.com/paper/2409.13347/) (2024)
- [mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual Reality](https://mlsanity.com/paper/2411.09914/) (2024)

## Abstract (arXiv)

Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.