# Video Test-Time Adaptation for Action Recognition

arXiv: https://arxiv.org/abs/2211.15393 · HTML: https://mlsanity.com/paper/2211.15393/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 57 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift](https://mlsanity.com/paper/2006.10963/) (2020)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [Test-Time Training with Masked Autoencoders](https://mlsanity.com/paper/2209.07522/) (2022)
- [Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning](https://mlsanity.com/paper/1702.08811/) (2017)

## Built on by (most cited first, 30 of 35)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization](https://mlsanity.com/paper/2311.01459/) (2023)
- [Test-Time Model Adaptation with Only Forward Passes](https://mlsanity.com/paper/2404.01650/) (2024)
- [Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2305.18010/) (2023)
- [Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models](https://mlsanity.com/paper/2403.12952/) (2024)
- [Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation](https://mlsanity.com/paper/2311.13209/) (2023)
- [Test-Time Zero-Shot Temporal Action Localization](https://mlsanity.com/paper/2404.05426/) (2024)
- [Vision transformers in domain adaptation and domain generalization: a study of robustness](https://mlsanity.com/paper/2404.04452/) (2024)
- [Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation](https://mlsanity.com/paper/2505.20704/) (2025)
- [Learning from Streaming Video with Orthogonal Gradients](https://mlsanity.com/paper/2504.01961/) (2025)
- [H-MoRe: Learning Human-centric Motion Representation for Action Analysis](https://mlsanity.com/paper/2504.10676/) (2025)
- [Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM](https://mlsanity.com/paper/2507.06973/) (2025)
- [Self-Bootstrapping for Versatile Test-Time Adaptation](https://mlsanity.com/paper/2504.08010/) (2025)
- [Source-free Video Domain Adaptation by Learning from Noisy Labels](https://mlsanity.com/paper/2311.18572/) (2023)
- [Fully Test-Time rPPG Estimation via Synthetic Signal-Guided Feature Learning](https://mlsanity.com/paper/2407.13322/) (2024)
- [TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models](https://mlsanity.com/paper/2403.11691/) (2024)
- [Efficient Diffusion-Driven Corruption Editor for Test-Time Adaptation](https://mlsanity.com/paper/2403.10911/) (2024)
- [Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models](https://mlsanity.com/paper/2501.13795/) (2025)
- [Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation](https://mlsanity.com/paper/2506.12481/) (2025)
- [Free-Form Composition Networks for Egocentric Action Recognition](https://mlsanity.com/paper/2307.06527/) (2023)
- [DATTA: Domain-Adversarial Test-Time Adaptation for Cross-Domain WiFi-Based Human Activity Recognition](https://mlsanity.com/paper/2411.13284/) (2024)
- [Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency](https://mlsanity.com/paper/2603.09798/) (2026)
- [CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition](https://mlsanity.com/paper/2603.27999/) (2026)
- [Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts](https://mlsanity.com/paper/2401.11406/) (2024)
- [EdgeSync: Faster Edge-model Updating via Adaptive Continuous Learning for Video Data Drift](https://mlsanity.com/paper/2406.03001/) (2024)
- [Augmented Contrastive Clustering with Uncertainty-Aware Prototyping for Time Series Test Time Adaptation](https://mlsanity.com/paper/2501.01472/) (2025)
- [Online Test-time Adaptation for 3D Human Pose Estimation: A Practical Perspective with Estimated 2D Poses](https://mlsanity.com/paper/2503.11194/) (2025)
- [Low-Cost Test-Time Adaptation for Robust Video Editing](https://mlsanity.com/paper/2507.21858/) (2025)
- [Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shifts](https://mlsanity.com/paper/2508.20488/) (2025)
- [Reasoning over the Behaviour of Objects in Video-Clips for Adverb-Type Recognition](https://mlsanity.com/paper/2307.04132/) (2023)

## Abstract (arXiv)

Although action recognition systems can achieve top performance when evaluated on in-distribution test points, they are vulnerable to unanticipated distribution shifts in test data. However, test-time adaptation of video action recognition models against common distribution shifts has so far not been demonstrated. We propose to address this problem with an approach tailored to spatio-temporal models that is capable of adaptation on a single video sample at a step. It consists in a feature distribution alignment technique that aligns online estimates of test set statistics towards the training statistics. We further enforce prediction consistency over temporally augmented views of the same test video sample. Evaluations on three benchmark action recognition datasets show that our proposed technique is architecture-agnostic and able to significantly boost the performance on both, the state of the art convolutional architecture TANet and the Video Swin Transformer. Our proposed method demonstrates a substantial performance gain over existing test-time adaptation approaches in both evaluations of a single distribution shift and the challenging case of random distribution shifts. Code will be available at urlhttps://github.com/wlin-at/ViTTA.