# Point3D: tracking actions as moving points with 3D CNNs

arXiv: https://arxiv.org/abs/2203.10584 · HTML: https://mlsanity.com/paper/2203.10584/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 30 of 40)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Actor-Centric Relation Network](https://mlsanity.com/paper/1807.10982/) (2018)
- [Action Tubelet Detector for Spatio-Temporal Action Localization](https://mlsanity.com/paper/1705.01861/) (2017)
- [Asynchronous Interaction Aggregation for Action Detection](https://mlsanity.com/paper/2004.07485/) (2020)
- [You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization](https://mlsanity.com/paper/1911.06644/) (2019)
- [Online Real-time Multiple Spatiotemporal Action Localisation and Prediction](https://mlsanity.com/paper/1611.08563/) (2016)
- [Actions as Moving Points](https://mlsanity.com/paper/2001.04608/) (2020)
- [STEP: Spatio-Temporal Progressive Learning for Video Action Detection](https://mlsanity.com/paper/1904.09288/) (2019)
- [Finding Action Tubes](https://mlsanity.com/paper/1411.6031/) (2014)
- [Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos](https://mlsanity.com/paper/1703.10664/) (2017)

## Built on by (most cited first, 1 of 1)

- [A Simple and Efficient Pipeline to Build an End-to-End Spatial-Temporal Action Detector](https://mlsanity.com/paper/2206.03064/) (2022)

## Abstract (arXiv)

Spatio-temporal action recognition has been a challenging task that involves detecting where and when actions occur. Current state-of-the-art action detectors are mostly anchor-based, requiring sensitive anchor designs and huge computations due to calculating large numbers of anchor boxes. Motivated by nascent anchor-free approaches, we propose Point3D, a flexible and computationally efficient network with high precision for spatio-temporal action recognition. Our Point3D consists of a Point Head for action localization and a 3D Head for action classification. Firstly, Point Head is used to track center points and knot key points of humans to localize the bounding box of an action. These location features are then piped into a time-wise attention to learn long-range dependencies across frames. The 3D Head is later deployed for the final action classification. Our Point3D achieves state-of-the-art performance on the JHMDB, UCF101-24, and AVA benchmarks in terms of frame-mAP and video-mAP. Comprehensive ablation studies also demonstrate the effectiveness of each module proposed in our Point3D.