# Efficient Online Multi-Person 2D Pose Tracking with Recurrent Spatio-Temporal Affinity Fields

arXiv: https://arxiv.org/abs/1811.11975 · HTML: https://mlsanity.com/paper/1811.11975/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2018.

## Built on by (most cited first, 30 of 32)

- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective](https://mlsanity.com/paper/2104.11536/) (2021)
- [SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos](https://mlsanity.com/paper/2104.11452/) (2021)
- [VoxelTrack: Multi-Person 3D Human Pose Estimation and Tracking in the Wild](https://mlsanity.com/paper/2108.02452/) (2021)
- [Learning Dynamics via Graph Neural Networks for Human Pose Estimation and Tracking](https://mlsanity.com/paper/2106.03772/) (2021)
- [DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2307.16687/) (2023)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [2D Human Pose Estimation: A Survey](https://mlsanity.com/paper/2204.07370/) (2022)
- [Rethinking Keypoint Representations: Modeling Keypoints and Poses as Objects for Multi-Person Human Pose Estimation](https://mlsanity.com/paper/2111.08557/) (2021)
- [Learning to Track Instances without Video Annotations](https://mlsanity.com/paper/2104.00287/) (2021)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction](https://mlsanity.com/paper/2109.00953/) (2021)
- [Joint-Motion Mutual Learning for Pose Estimation in Videos](https://mlsanity.com/paper/2408.02285/) (2024)
- [Snipper: A Spatiotemporal Transformer for Simultaneous Multi-Person 3D Pose Estimation Tracking and Forecasting on a Video Snippet](https://mlsanity.com/paper/2207.04320/) (2022)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [Capturing Detailed Deformations of Moving Human Bodies](https://mlsanity.com/paper/2102.07343/) (2021)
- [Human Modelling and Pose Estimation Overview](https://mlsanity.com/paper/2406.19290/) (2024)
- [Learning to Estimate External Forces of Human Motion in Video](https://mlsanity.com/paper/2207.05845/) (2022)
- [A Gated Attention Transformer for Multi-Person Pose Tracking](https://mlsanity.com/paper/2306.05807/) (2023)
- [Improving Multi-Person Pose Tracking with A Confidence Network](https://mlsanity.com/paper/2310.18920/) (2023)
- [DenseTrack: Drone-based Crowd Tracking via Density-aware Motion-appearance Synergy](https://mlsanity.com/paper/2407.17272/) (2024)
- [Temporally Guided Articulated Hand Pose Tracking in Surgical Videos](https://mlsanity.com/paper/2101.04281/) (2021)
- [A Review on Human Pose Estimation](https://mlsanity.com/paper/2110.06877/) (2021)
- [Camera Motion Agnostic 3D Human Pose Estimation](https://mlsanity.com/paper/2112.00343/) (2021)
- [Koopman pose predictions for temporally consistent human walking estimations](https://mlsanity.com/paper/2205.02737/) (2022)
- [People Tracking and Re-Identifying in Distributed Contexts: Extension Study of PoseTReID](https://mlsanity.com/paper/2205.10086/) (2022)
- [ErgoExplorer: Interactive Ergonomic Risk Assessment from Video Collections](https://mlsanity.com/paper/2209.05252/) (2022)
- [Augmenting Vision-Based Human Pose Estimation with Rotation Matrix](https://mlsanity.com/paper/2310.06068/) (2023)

## Abstract (arXiv)

We present an online approach to efficiently and simultaneously detect and track the 2D pose of multiple people in a video sequence. We build upon Part Affinity Field (PAF) representation designed for static images, and propose an architecture that can encode and predict Spatio-Temporal Affinity Fields (STAF) across a video sequence. In particular, we propose a novel temporal topology cross-linked across limbs which can consistently handle body motions of a wide range of magnitudes. Additionally, we make the overall approach recurrent in nature, where the network ingests STAF heatmaps from previous frames and estimates those for the current frame. Our approach uses only online inference and tracking, and is currently the fastest and the most accurate bottom-up approach that is runtime invariant to the number of people in the scene and accuracy invariant to input frame rate of camera. Running at ~30 fps on a single GPU at single scale, it achieves highly competitive results on the PoseTrack benchmarks.