# Self-Supervised Learning via multi-Transformation Classification for Action Recognition

arXiv: https://arxiv.org/abs/2102.10378 · HTML: https://mlsanity.com/paper/2102.10378/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 20 Feb 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Multi-task Self-Supervised Visual Learning](https://mlsanity.com/paper/1708.07860/) (2017)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles](https://mlsanity.com/paper/1811.09795/) (2018)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics](https://mlsanity.com/paper/1904.03597/) (2019)
- [Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition](https://mlsanity.com/paper/1808.07507/) (2018)
- [Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning](https://mlsanity.com/paper/1807.11293/) (2018)
- [Cross and Learn: Cross-Modal Self-Supervision](https://mlsanity.com/paper/1811.03879/) (2018)
- [Self-supervised Learning for ECG-based Emotion Recognition](https://mlsanity.com/paper/1910.07497/) (2019)
- [Evolving Space-Time Neural Architectures for Videos](https://mlsanity.com/paper/1811.10636/) (2018)
- [Skip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking](https://mlsanity.com/paper/1910.12770/) (2019)

## Built on by (most cited first, 5 of 5)

- [CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2212.05136/) (2022)
- [Contextual Explainable Video Representation: Human Perception-based Understanding](https://mlsanity.com/paper/2212.06206/) (2022)
- [Multimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised Learning](https://mlsanity.com/paper/2210.06297/) (2022)
- [An Effective End-to-End Solution for Multimodal Action Recognition](https://mlsanity.com/paper/2506.09345/) (2025)
- [DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition](https://mlsanity.com/paper/2507.12426/) (2025)

## Abstract (arXiv)

Self-supervised tasks have been utilized to build useful representations that can be used in downstream tasks when the annotation is unavailable. In this paper, we introduce a self-supervised video representation learning method based on the multi-transformation classification to efficiently classify human actions. Self-supervised learning on various transformations not only provides richer contextual information but also enables the visual representation more robust to the transforms. The spatio-temporal representation of the video is learned in a self-supervised manner by classifying seven different transformations i.e. rotation, clip inversion, permutation, split, join transformation, color switch, frame replacement, noise addition. First, seven different video transformations are applied to video clips. Then the 3D convolutional neural networks are utilized to extract features for clips and these features are processed to classify the pseudo-labels. We use the learned models in pretext tasks as the pre-trained models and fine-tune them to recognize human actions in the downstream task. We have conducted the experiments on UCF101 and HMDB51 datasets together with C3D and 3D Resnet-18 as backbone networks. The experimental results have shown that our proposed framework is outperformed other SOTA self-supervised action recognition approaches. The code will be made publicly available.