# DeePoint: Visual Pointing Recognition and Direction Estimation

arXiv: https://arxiv.org/abs/2304.06977 · HTML: https://mlsanity.com/paper/2304.06977/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network](https://mlsanity.com/paper/2203.14564/) (2022)
- [OadTR: Online Action Detection with Transformers](https://mlsanity.com/paper/2106.11149/) (2021)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [Commodifying Pointing in HRI: Simple and Fast Pointing Gesture Detection from RGB-D Images](https://mlsanity.com/paper/1902.02636/) (2019)
- [Recognition and Localisation of Pointing Gestures using a RGB-D Camera](https://mlsanity.com/paper/2001.03687/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions](https://mlsanity.com/paper/2403.17545/) (2024)
- [Omni-MMSI: Toward Identity-attributed Social Interaction Understanding](https://mlsanity.com/paper/2604.00267/) (2026)
- [DABACO: A Multi-Camera Dataset and Benchmark for Screen Localization and Pointing Estimation](https://mlsanity.com/paper/2610.03928/) (2026)

## Abstract (arXiv)

In this paper, we realize automatic visual recognition and direction estimation of pointing. We introduce the first neural pointing understanding method based on two key contributions. The first is the introduction of a first-of-its-kind large-scale dataset for pointing recognition and direction estimation, which we refer to as the DP Dataset. DP Dataset consists of more than 2 million frames of 33 people pointing in various styles annotated for each frame with pointing timings and 3D directions. The second is DeePoint, a novel deep network model for joint recognition and 3D direction estimation of pointing. DeePoint is a Transformer-based network which fully leverages the spatio-temporal coordination of the body parts, not just the hands. Through extensive experiments, we demonstrate the accuracy and efficiency of DeePoint. We believe DP Dataset and DeePoint will serve as a sound foundation for visual human intention understanding.