# Learning Dynamics via Graph Neural Networks for Human Pose Estimation and Tracking

arXiv: https://arxiv.org/abs/2106.03772 · HTML: https://mlsanity.com/paper/2106.03772/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [PoseTrack: A Benchmark for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1710.10000/) (2017)
- [SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels](https://mlsanity.com/paper/1711.08920/) (2017)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation](https://mlsanity.com/paper/1511.06645/) (2015)
- [DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model](https://mlsanity.com/paper/1605.03170/) (2016)
- [Pose Flow: Efficient Online Pose Tracking](https://mlsanity.com/paper/1802.00977/) (2018)
- [Factorizable Graph Convolutional Networks](https://mlsanity.com/paper/2010.05421/) (2020)
- [Dynamic Graph Message Passing Networks](https://mlsanity.com/paper/1908.06955/) (2019)
- [Flowing ConvNets for Human Pose Estimation in Videos](https://mlsanity.com/paper/1506.02897/) (2015)
- [Detect-and-Track: Efficient Pose Estimation in Videos](https://mlsanity.com/paper/1712.09184/) (2017)
- [Combining detection and tracking for human pose estimation in videos](https://mlsanity.com/paper/2003.13743/) (2020)

## Built on by (most cited first, 30 of 37)

- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Human Pose as Compositional Tokens](https://mlsanity.com/paper/2303.11638/) (2023)
- [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023)
- [DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2307.16687/) (2023)
- [Graph Matching with Bi-level Noisy Correspondence](https://mlsanity.com/paper/2212.04085/) (2022)
- [Learning Graph Neural Networks for Image Style Transfer](https://mlsanity.com/paper/2207.11681/) (2022)
- [Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks](https://mlsanity.com/paper/2109.12872/) (2021)
- [PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision](https://mlsanity.com/paper/2203.15625/) (2022)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [2D Human Pose Estimation: A Survey](https://mlsanity.com/paper/2204.07370/) (2022)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [Coverage Axis++: Efficient Inner Point Selection for 3D Shape Skeletonization](https://mlsanity.com/paper/2401.12946/) (2024)
- [Video-Based Human Pose Regression via Decoupled Space-Time Aggregation](https://mlsanity.com/paper/2403.19926/) (2024)
- [Self-Correctable and Adaptable Inference for Generalizable Human Pose Estimation](https://mlsanity.com/paper/2303.11180/) (2023)
- [Joint-Motion Mutual Learning for Pose Estimation in Videos](https://mlsanity.com/paper/2408.02285/) (2024)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [Category-Agnostic 6D Pose Estimation with Conditional Neural Processes](https://mlsanity.com/paper/2206.07162/) (2022)
- [Self-Constrained Inference Optimization on Structural Groups for Human Pose Estimation](https://mlsanity.com/paper/2207.02425/) (2022)
- [A Differential Geometric View and Explainability of GNN on Evolving Graphs](https://mlsanity.com/paper/2403.06425/) (2024)
- [High-Resolution Spatiotemporal Modeling with Global-Local State Space Models for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2510.11017/) (2025)
- [A Gated Attention Transformer for Multi-Person Pose Tracking](https://mlsanity.com/paper/2306.05807/) (2023)
- [Improving Multi-Person Pose Tracking with A Confidence Network](https://mlsanity.com/paper/2310.18920/) (2023)
- [Free-Form Composition Networks for Egocentric Action Recognition](https://mlsanity.com/paper/2307.06527/) (2023)
- [GoalieNet: A Multi-Stage Network for Joint Goalie, Equipment, and Net Pose Estimation in Ice Hockey](https://mlsanity.com/paper/2306.15853/) (2023)
- [Waterfall Transformer for Multi-person Pose Estimation](https://mlsanity.com/paper/2411.18944/) (2024)
- [Graph Neural Networks for Jamming Source Localization](https://mlsanity.com/paper/2506.03196/) (2025)
- [CIGPose: Causal Intervention Graph Neural Network for Whole-Body Pose Estimation](https://mlsanity.com/paper/2603.09418/) (2026)
- [Explaining GNN over Evolving Graphs using Information Flow](https://mlsanity.com/paper/2111.10037/) (2021)
- [MapTrack: Tracking in the Map](https://mlsanity.com/paper/2402.12968/) (2024)

## Abstract (arXiv)

Multi-person pose estimation and tracking serve as crucial steps for video understanding. Most state-of-the-art approaches rely on first estimating poses in each frame and only then implementing data association and refinement. Despite the promising results achieved, such a strategy is inevitably prone to missed detections especially in heavily-cluttered scenes, since this tracking-by-detection paradigm is, by nature, largely dependent on visual evidences that are absent in the case of occlusion. In this paper, we propose a novel online approach to learning the pose dynamics, which are independent of pose detections in current fame, and hence may serve as a robust estimation even in challenging scenarios including occlusion. Specifically, we derive this prediction of dynamics through a graph neural network~(GNN) that explicitly accounts for both spatial-temporal and visual information. It takes as input the historical pose tracklets and directly predicts the corresponding poses in the following frame for each tracklet. The predicted poses will then be aggregated with the detected poses, if any, at the same frame so as to produce the final pose, potentially recovering the occluded joints missed by the estimator. Experiments on PoseTrack 2017 and PoseTrack 2018 datasets demonstrate that the proposed method achieves results superior to the state of the art on both human pose estimation and tracking tasks.