# Learning from Temporal Gradient for Semi-supervised Action Recognition

arXiv: https://arxiv.org/abs/2111.13241 · HTML: https://mlsanity.com/paper/2111.13241/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 67 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 30 of 50)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)

## Built on by (most cited first, 26 of 26)

- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [SVFormer: Semi-supervised Video Transformer for Action Recognition](https://mlsanity.com/paper/2211.13222/) (2022)
- [UniMatch V2: Pushing the Limit of Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2410.10777/) (2024)
- [Masked Autoencoders Enable Efficient Knowledge Distillers](https://mlsanity.com/paper/2208.12256/) (2022)
- [MMAD: Multi-label Micro-Action Detection in Videos](https://mlsanity.com/paper/2407.05311/) (2024)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition](https://mlsanity.com/paper/2303.16268/) (2023)
- [Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection](https://mlsanity.com/paper/2303.17285/) (2023)
- [X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2312.07378/) (2023)
- [ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos](https://mlsanity.com/paper/2208.07929/) (2022)
- [Video Recognition in Portrait Mode](https://mlsanity.com/paper/2312.13746/) (2023)
- [Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosis](https://mlsanity.com/paper/2304.10073/) (2023)
- [Distillation-guided Representation Learning for Unconstrained Gait Recognition](https://mlsanity.com/paper/2307.14578/) (2023)
- [ActNetFormer: Transformer-ResNet Hybrid Method for Semi-Supervised Action Recognition in Videos](https://mlsanity.com/paper/2404.06243/) (2024)
- [FinePseudo: Improving Pseudo-Labelling through Temporal-Alignablity for Semi-Supervised Fine-Grained Action Recognition](https://mlsanity.com/paper/2409.01448/) (2024)
- [Dual Invariance Self-training for Reliable Semi-supervised Surgical Phase Recognition](https://mlsanity.com/paper/2501.17628/) (2025)
- [On the Surprising Effectiveness of Transformers in Low-Labeled Video Recognition](https://mlsanity.com/paper/2209.07474/) (2022)
- [DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition](https://mlsanity.com/paper/2507.12426/) (2025)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [SITAR: Semi-supervised Image Transformer for Action Recognition](https://mlsanity.com/paper/2409.02910/) (2024)
- [Towards Micro-Action Recognition with Limited Annotations: An Asynchronous Pseudo Labeling and Training Approach](https://mlsanity.com/paper/2504.07785/) (2025)
- [Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learning](https://mlsanity.com/paper/2509.06306/) (2025)
- [3D Dynamic Radio Map Prediction Using Vision Transformers for Low-Altitude Wireless Networks](https://mlsanity.com/paper/2511.19019/) (2025)
- [Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup](https://mlsanity.com/paper/2503.02284/) (2025)
- [A Review on Coarse to Fine-Grained Animal Action Recognition](https://mlsanity.com/paper/2506.01214/) (2025)
- [LEViL: Label-Efficient Video Learning via Zero-Shot Distillation over VLM-Generated Pseudo-Label Spaces](https://mlsanity.com/paper/2606.21358/) (2026)

## Abstract (arXiv)

Semi-supervised video action recognition tends to enable deep neural networks to achieve remarkable performance even with very limited labeled data. However, existing methods are mainly transferred from current image-based methods (e.g., FixMatch). Without specifically utilizing the temporal dynamics and inherent multimodal attributes, their results could be suboptimal. To better leverage the encoded temporal information in videos, we introduce temporal gradient as an additional modality for more attentive feature extraction in this paper. To be specific, our method explicitly distills the fine-grained motion representations from temporal gradient (TG) and imposes consistency across different modalities (i.e., RGB and TG). The performance of semi-supervised action recognition is significantly improved without additional computation or parameters during inference. Our method achieves the state-of-the-art performance on three video action recognition benchmarks (i.e., Kinetics-400, UCF-101, and HMDB-51) under several typical semi-supervised settings (i.e., different ratios of labeled data).