mlsanityData to 7 Oct 2026

Transformers in Action Recognition: A Review on Temporal Modeling

arXiv 2302.01921 ↗First posted 29 Dec 2022cs.CV cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In vision-based action recognition, spatio-temporal features from different modalities are used for recognizing activities. Temporal modeling is a long challenge of action recognition. However, there are limited methods such as pre-computed motion features, three-dimensional (3D) filters, and recurrent neural networks (RNN) for modeling motion information in deep-based approaches. Recently, transformers success in modeling long-range dependencies in natural language processing (NLP) tasks has gotten great attention from other domains; including speech, image, and video, to rely entirely on self-attention without using sequence-aligned RNNs or convolutions. Although the application of transformers to action recognition is relatively new, the amount of research proposed on this topic within the last few years is astounding. This paper especially reviews recent progress in deep learning methods for modeling temporal variations. It focuses on action recognition methods that use transformers for temporal modeling, discussing their main features, used modalities, and identifying opportunities and challenges for future research.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 149 of its 294 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

149 of 294 references.

References · 149

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  14. 2015Spatial Transformer Networkscited by 1,423
  15. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  16. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  17. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  18. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  19. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  20. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  21. 2021Video Swin Transformercited by 1,090
  22. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  23. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  24. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  25. 2021Transformers in Vision: A Surveycited by 1,009
  26. 2021Multiscale Vision Transformerscited by 896
  27. 2020Pre-Trained Image Processing Transformercited by 891
  28. 2019Are Sixteen Heads Really Better than One?cited by 803
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  31. 2020A Survey on Visual Transformercited by 653
  32. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  33. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  34. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  35. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  36. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  37. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  38. 2020End-to-End Video Instance Segmentation with Transformerscited by 443

50 most cited of 149 All 149 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligencecited by 25
  2. 2024Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modelingcited by 4
  3. 2026Attention-Guided Dual-Stream Learning for Group Engagement Recognition: Fusing Transformer-Encoded Motion Dynamics with Scene Context via Adaptive Gatingcited by 0