# Human keypoint detection for close proximity human-robot interaction

arXiv: https://arxiv.org/abs/2207.07742 · HTML: https://mlsanity.com/paper/2207.07742/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 15 Jul 2022.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [Whole-Body Human Pose Estimation in the Wild](https://mlsanity.com/paper/2007.11858/) (2020)
- [PoseTrack: A Benchmark for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1710.10000/) (2017)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Pose Flow: Efficient Online Pose Tracking](https://mlsanity.com/paper/1802.00977/) (2018)
- [PaStaNet: Toward Human Activity Knowledge Engine](https://mlsanity.com/paper/2004.00945/) (2020)
- [Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation](https://mlsanity.com/paper/1707.05388/) (2017)
- [3D Human Pose Estimation in RGBD Images for Robotic Task Learning](https://mlsanity.com/paper/1803.02622/) (2018)
- [Safe physical HRI: Toward a unified treatment of speed and separation monitoring together with power and force limiting](https://mlsanity.com/paper/1908.03046/) (2019)
- [Compact Real-time avoidance on a Humanoid Robot for Human-robot Interaction](https://mlsanity.com/paper/1801.05671/) (2018)
- [Toward safe separation distance monitoring from RGB-D sensors in human-robot interaction](https://mlsanity.com/paper/1810.04953/) (2018)

## Built on by (most cited first, 2 of 2)

- [Testing Human-Hand Segmentation on In-Distribution and Out-of-Distribution Data in Human-Robot Interactions Using a Deep Ensemble Model](https://mlsanity.com/paper/2501.07713/) (2025)
- [A Study on Real-time Object Detection using Deep Learning](https://mlsanity.com/paper/2602.15926/) (2026)

## Abstract (arXiv)

We study the performance of state-of-the-art human keypoint detectors in the context of close proximity human-robot interaction. The detection in this scenario is specific in that only a subset of body parts such as hands and torso are in the field of view. In particular, (i) we survey existing datasets with human pose annotation from the perspective of close proximity images and prepare and make publicly available a new Human in Close Proximity (HiCP) dataset; (ii) we quantitatively and qualitatively compare state-of-the-art human whole-body 2D keypoint detection methods (OpenPose, MMPose, AlphaPose, Detectron2) on this dataset; (iii) since accurate detection of hands and fingers is critical in applications with handovers, we evaluate the performance of the MediaPipe hand detector; (iv) we deploy the algorithms on a humanoid robot with an RGB-D camera on its head and evaluate the performance in 3D human keypoint detection. A motion capture system is used as reference. The best performing whole-body keypoint detectors in close proximity were MMPose and AlphaPose, but both had difficulty with finger detection. Thus, we propose a combination of MMPose or AlphaPose for the body and MediaPipe for the hands in a single framework providing the most accurate and robust detection. We also analyse the failure modes of individual detectors -- for example, to what extent the absence of the head of the person in the image degrades performance. Finally, we demonstrate the framework in a scenario where a humanoid robot interacting with a person uses the detected 3D keypoints for whole-body avoidance maneuvers.