# Can An Image Classifier Suffice For Action Recognition?

arXiv: https://arxiv.org/abs/2106.14104 · HTML: https://mlsanity.com/paper/2106.14104/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 60 references that are in the map.
- First posted on arXiv 26 Jun 2021.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)

## Built on by (most cited first, 20 of 20)

- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [TALL: Thumbnail Layout for Deepfake Video Detection](https://mlsanity.com/paper/2307.07494/) (2023)
- [Time Series as Images: Vision Transformer for Irregularly Sampled Time Series](https://mlsanity.com/paper/2303.12799/) (2023)
- [Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring](https://mlsanity.com/paper/2301.11116/) (2023)
- [Memory Consolidation Enables Long-Context Video Understanding](https://mlsanity.com/paper/2402.05861/) (2024)
- [Dual-path Adaptation from Image to Video Transformers](https://mlsanity.com/paper/2303.09857/) (2023)
- [ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video](https://mlsanity.com/paper/2310.01324/) (2023)
- [Video Panels for Long Video Understanding](https://mlsanity.com/paper/2509.23724/) (2025)
- [Vision-Language Models Learn Super Images for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2312.00414/) (2023)
- [VideoPipe 2022 Challenge: Real-World Video Understanding for Urban Pipe Inspection](https://mlsanity.com/paper/2210.11158/) (2022)
- [Extending Video Masked Autoencoders to 128 frames](https://mlsanity.com/paper/2411.13683/) (2024)
- [Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding](https://mlsanity.com/paper/2602.18977/) (2026)
- [SimLVSeg: Simplifying Left Ventricular Segmentation in 2D+Time Echocardiograms with Self- and Weakly-Supervised Learning](https://mlsanity.com/paper/2310.00454/) (2023)
- [No Time to Waste: Squeeze Time into Channel for Mobile Video Understanding](https://mlsanity.com/paper/2405.08344/) (2024)
- [SITAR: Semi-supervised Image Transformer for Action Recognition](https://mlsanity.com/paper/2409.02910/) (2024)
- [MamBOA: State-Space Architecture for Video Recognition](https://mlsanity.com/paper/2606.15275/) (2026)
- [Super Images -- A New 2D Perspective on 3D Medical Imaging Analysis](https://mlsanity.com/paper/2205.02847/) (2022)
- [Diagnosis and Prognosis of Head and Neck Cancer Patients using Artificial Intelligence](https://mlsanity.com/paper/2306.00034/) (2023)
- [Joint learning of images and videos with a single Vision Transformer](https://mlsanity.com/paper/2308.10533/) (2023)
- [Image Classifiers are Efficient Self-Supervised Video Representation Learners](https://mlsanity.com/paper/2609.40347/) (2026)

## Abstract (arXiv)

We explore a new perspective on video understanding by casting the video recognition problem as an image recognition task. Our approach rearranges input video frames into super images, which allow for training an image classifier directly to fulfill the task of action recognition, in exactly the same way as image classification. With such a simple idea, we show that transformer-based image classifiers alone can suffice for action recognition. In particular, our approach demonstrates strong and promising performance against SOTA methods on several public datasets including Kinetics400, Moments In Time, Something-Something V2 (SSV2), Jester and Diving48. We also experiment with the prevalent ResNet image classifiers in computer vision to further validate our idea. The results on both Kinetics400 and SSV2 are comparable to some of the best-performed CNN approaches based on spatio-temporal modeling. Our source codes and models are available at https://github.com/IBM/sifar-pytorch.