# Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video

arXiv: https://arxiv.org/abs/2502.10616 · HTML: https://mlsanity.com/paper/2502.10616/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [Learning Delicate Local Representations for Multi-Person Pose Estimation](https://mlsanity.com/paper/2003.04030/) (2020)
- [ViTPose++: Vision Transformer for Generic Body Pose Estimation](https://mlsanity.com/paper/2212.04246/) (2022)
- [Learning Temporal Pose Estimation from Sparsely-Labeled Videos](https://mlsanity.com/paper/1906.04016/) (2019)
- [Self-supervised Keypoint Correspondences for Multi-Person Pose Estimation and Tracking in Videos](https://mlsanity.com/paper/2004.12652/) (2020)

## Abstract (arXiv)

Temporal modeling and spatio-temporal collaboration are pivotal techniques for video-based human pose estimation. Most state-of-the-art methods adopt optical flow or temporal difference, learning local visual content correspondence across frames at the pixel level, to capture motion dynamics. However, such a paradigm essentially relies on localized pixel-to-pixel similarity, which neglects the semantical correlations among frames and is vulnerable to image quality degradations (e.g. occlusions or blur). Moreover, existing approaches often combine motion and spatial (appearance) features via simple concatenation or summation, leading to practical challenges in fully leveraging these distinct modalities. In this paper, we present a novel framework that learns multi-level semantical dynamics and dense spatio-temporal collaboration for multi-frame human pose estimation. Specifically, we first design a Multi-Level Semantic Motion Encoder using a multi-masked context and pose reconstruction strategy. This strategy stimulates the model to explore multi-granularity spatiotemporal semantic relationships among frames by progressively masking the features of (patch) cubes and frames. We further introduce a Spatial-Motion Mutual Learning module which densely propagates and consolidates context information from spatial and motion features to enhance the capability of the model. Extensive experiments demonstrate that our approach sets new state-of-the-art results on three benchmark datasets, PoseTrack2017, PoseTrack2018, and PoseTrack21.