# OCHID-Fi: Occlusion-Robust Hand Pose Estimation in 3D via RF-Vision

arXiv: https://arxiv.org/abs/2308.10146 · HTML: https://mlsanity.com/paper/2308.10146/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Deep Complex Networks](https://mlsanity.com/paper/1705.09792/) (2017)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [3D Hand Shape and Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1903.00812/) (2019)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Hand Pose Estimation via Latent 2.5D Heatmap Regression](https://mlsanity.com/paper/1804.09534/) (2018)
- [Cross-modal Deep Variational Hand Pose Estimation](https://mlsanity.com/paper/1803.11404/) (2018)
- [Person-in-WiFi: Fine-grained Person Perception using WiFi](https://mlsanity.com/paper/1904.00276/) (2019)
- [Occlusion Robust Face Recognition Based on Mask Learning with PairwiseDifferential Siamese Network](https://mlsanity.com/paper/1908.06290/) (2019)
- [RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects](https://mlsanity.com/paper/2007.14366/) (2020)
- [A neural network based on SPD manifold learning for skeleton-based hand gesture recognition](https://mlsanity.com/paper/1904.12970/) (2019)
- [Distilling Knowledge From a Deep Pose Regressor Network](https://mlsanity.com/paper/1908.00858/) (2019)
- [Regressive Domain Adaptation for Unsupervised Keypoint Detection](https://mlsanity.com/paper/2103.06175/) (2021)
- [Multi-View Radar Semantic Segmentation](https://mlsanity.com/paper/2103.16214/) (2021)
- [RF-Net: a Unified Meta-learning Framework for RF-enabled One-shot Human Activity Recognition](https://mlsanity.com/paper/2111.04566/) (2021)
- [Orthogonal Deep Features Decomposition for Age-Invariant Face Recognition](https://mlsanity.com/paper/1810.07599/) (2018)
- [PointAR: Efficient Lighting Estimation for Mobile Augmented Reality](https://mlsanity.com/paper/2004.00006/) (2020)
- [CoupleNet: Coupling Global Structure with Local Parts for Object Detection](https://mlsanity.com/paper/1708.02863/) (2017)
- [HandFoldingNet: A 3D Hand Pose Estimation Network Using Multiscale-Feature Guided Folding of a 2D Hand Skeleton](https://mlsanity.com/paper/2108.05545/) (2021)
- [RF-Based Human Activity Recognition Using Signal Adapted Convolutional Neural Network](https://mlsanity.com/paper/2110.14307/) (2021)
- [SiWa: See into Walls via Deep UWB Radar](https://mlsanity.com/paper/2110.14279/) (2021)
- [Efficient Convolutional Neural Network for FMCW Radar Based Hand Gesture Recognition](https://mlsanity.com/paper/1908.10560/) (2019)

## Built on by (most cited first, 2 of 2)

- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [Radio Frequency Signal based Human Silhouette Segmentation: A Sequential Diffusion Approach](https://mlsanity.com/paper/2407.19244/) (2024)

## Abstract (arXiv)

Hand Pose Estimation (HPE) is crucial to many applications, but conventional cameras-based CM-HPE methods are completely subject to Line-of-Sight (LoS), as cameras cannot capture occluded objects. In this paper, we propose to exploit Radio-Frequency-Vision (RF-vision) capable of bypassing obstacles for achieving occluded HPE, and we introduce OCHID-Fi as the first RF-HPE method with 3D pose estimation capability. OCHID-Fi employs wideband RF sensors widely available on smart devices (e.g., iPhones) to probe 3D human hand pose and extract their skeletons behind obstacles. To overcome the challenge in labeling RF imaging given its human incomprehensible nature, OCHID-Fi employs a cross-modality and cross-domain training process. It uses a pre-trained CM-HPE network and a synchronized CM/RF dataset, to guide the training of its complex-valued RF-HPE network under LoS conditions. It further transfers knowledge learned from labeled LoS domain to unlabeled occluded domain via adversarial learning, enabling OCHID-Fi to generalize to unseen occluded scenarios. Experimental results demonstrate the superiority of OCHID-Fi: it achieves comparable accuracy to CM-HPE under normal conditions while maintaining such accuracy even in occluded scenarios, with empirical evidence for its generalizability to new domains.