# Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation

arXiv: https://arxiv.org/abs/2203.15227 · HTML: https://mlsanity.com/paper/2203.15227/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 72 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 62)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation](https://mlsanity.com/paper/1908.10357/) (2019)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [PoseTrack: A Benchmark for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1710.10000/) (2017)
- [Distribution-Aware Coordinate Representation for Human Pose Estimation](https://mlsanity.com/paper/1910.06278/) (2019)
- [Semantic Flow for Fast and Accurate Scene Parsing](https://mlsanity.com/paper/2002.10120/) (2020)
- [PifPaf: Composite Fields for Human Pose Estimation](https://mlsanity.com/paper/1903.06593/) (2019)
- [Indices Matter: Learning to Index for Deep Image Matting](https://mlsanity.com/paper/1908.00672/) (2019)
- [Deconfounded Video Moment Retrieval with Causal Intervention](https://mlsanity.com/paper/2106.01534/) (2021)
- [FaPN: Feature-aligned Pyramid Network for Dense Image Prediction](https://mlsanity.com/paper/2108.07058/) (2021)
- [Pose Flow: Efficient Online Pose Tracking](https://mlsanity.com/paper/1802.00977/) (2018)
- [The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation](https://mlsanity.com/paper/1911.07524/) (2019)
- [Flowing ConvNets for Human Pose Estimation in Videos](https://mlsanity.com/paper/1506.02897/) (2015)
- [AlignSeg: Feature-Aligned Segmentation Networks](https://mlsanity.com/paper/2003.00872/) (2020)
- [Detect-and-Track: Efficient Pose Estimation in Videos](https://mlsanity.com/paper/1712.09184/) (2017)

## Built on by (most cited first, 24 of 24)

- [PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2404.05231/) (2024)
- [DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2307.16687/) (2023)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling](https://mlsanity.com/paper/2208.10211/) (2022)
- [Dynamic Multimodal Information Bottleneck for Multimodality Classification](https://mlsanity.com/paper/2311.01066/) (2023)
- [Copy Motion From One to Another: Fake Motion Video Generation](https://mlsanity.com/paper/2205.01373/) (2022)
- [Video-Based Human Pose Regression via Decoupled Space-Time Aggregation](https://mlsanity.com/paper/2403.19926/) (2024)
- [Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization](https://mlsanity.com/paper/2306.07576/) (2023)
- [REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer](https://mlsanity.com/paper/2209.00475/) (2022)
- [MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition](https://mlsanity.com/paper/2303.02575/) (2023)
- [Humans as Checkerboards: Calibrating Camera Motion Scale for World-Coordinate Human Mesh Recovery](https://mlsanity.com/paper/2407.00574/) (2024)
- [High-Resolution Spatiotemporal Modeling with Global-Local State Space Models for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2510.11017/) (2025)
- [Kinematics Modeling Network for Video-based Human Pose Estimation](https://mlsanity.com/paper/2207.10971/) (2022)
- [Learning to Agree on Vision Attention for Visual Commonsense Reasoning](https://mlsanity.com/paper/2302.02117/) (2023)
- [Iterative Graph Filtering Network for 3D Human Pose Estimation](https://mlsanity.com/paper/2307.16074/) (2023)
- [DINTR: Tracking via Diffusion-based Interpolation](https://mlsanity.com/paper/2410.10053/) (2024)
- [Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking](https://mlsanity.com/paper/2412.20002/) (2024)
- [End-to-End Multi-Person Pose Estimation with Pose-Aware Video Transformer](https://mlsanity.com/paper/2511.13208/) (2025)
- [Video Infringement Detection via Feature Disentanglement and Mutual Information Maximization](https://mlsanity.com/paper/2309.06877/) (2023)
- [Multi-Grained Feature Pruning for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2503.05365/) (2025)
- [An End-to-End Framework for Video Multi-Person Pose Estimation](https://mlsanity.com/paper/2509.01095/) (2025)
- [Beyond Static Frames: Temporal Aggregate-and-Restore Vision Transformer for Human Pose Estimation](https://mlsanity.com/paper/2603.05929/) (2026)
- [NCSTR: Node-Centric Decoupled Spatio-Temporal Reasoning for Video-based Human Pose Estimation](https://mlsanity.com/paper/2603.20323/) (2026)

## Abstract (arXiv)

Multi-frame human pose estimation has long been a compelling and fundamental problem in computer vision. This task is challenging due to fast motion and pose occlusion that frequently occur in videos. State-of-the-art methods strive to incorporate additional visual evidences from neighboring frames (supporting frames) to facilitate the pose estimation of the current frame (key frame). One aspect that has been obviated so far, is the fact that current methods directly aggregate unaligned contexts across frames. The spatial-misalignment between pose features of the current frame and neighboring frames might lead to unsatisfactory results. More importantly, existing approaches build upon the straightforward pose estimation loss, which unfortunately cannot constrain the network to fully leverage useful information from neighboring frames. To tackle these problems, we present a novel hierarchical alignment framework, which leverages coarse-to-fine deformations to progressively update a neighboring frame to align with the current frame at the feature level. We further propose to explicitly supervise the knowledge extraction from neighboring frames, guaranteeing that useful complementary cues are extracted. To achieve this goal, we theoretically analyzed the mutual information between the frames and arrived at a loss that maximizes the task-relevant mutual information. These allow us to rank No.1 in the Multi-frame Person Pose Estimation Challenge on benchmark dataset PoseTrack2017, and obtain state-of-the-art performance on benchmarks Sub-JHMDB and Pose-Track2018. Our code is released at https://github. com/Pose-Group/FAMI-Pose, hoping that it will be useful to the community.