mlsanityData to 7 Oct 2026

ActionFormer: Localizing Moments of Actions with Transformers

arXiv 2202.07925 ↗First posted 16 Feb 2022cs.CVcited by 308

Not on the map. Why?

Abstract (arXiv)

Self-attention based Transformer models have demonstrated impressive results for image classification and object detection, and more recently for video understanding. Inspired by this success, we investigate the application of Transformer networks for temporal action localization in videos. To this end, we present ActionFormer -- a simple yet powerful model to identify actions in time and recognize their categories in a single shot, without using action proposals or relying on pre-defined anchor windows. ActionFormer combines a multiscale feature representation with local self-attention, and uses a light-weighted decoder to classify every moment in time and estimate the corresponding action boundaries. We show that this orchestrated design results in major improvements upon prior works. Without bells and whistles, ActionFormer achieves 71.0% mAP at tIoU=0.5 on THUMOS14, outperforming the best prior model by 14.1 absolute percentage points. Further, ActionFormer demonstrates strong results on ActivityNet 1.3 (36.6% average mAP) and EPIC-Kitchens 100 (+13.5% average mAP over prior works). Our code is available at http://github.com/happyharrycn/actionformer_release.

Key facts. Cited by 308 papers in the map (data through 7 Oct 2026). Builds on 78 of its 91 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

78 of 91 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  13. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  14. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  15. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  16. 2021Video Swin Transformercited by 1,090
  17. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  18. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  19. 2021Multiscale Vision Transformerscited by 896
  20. 2021Going deeper with Image Transformerscited by 759
  21. 2019CenterNet: Keypoint Triplets for Object Detectioncited by 654
  22. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  23. 2019Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selectioncited by 539
  24. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  25. 2019Distance-IoU Loss: Faster and Better Learning for Bounding Box Regressioncited by 454
  26. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  27. 2021Early Convolutions Help Transformers See Bettercited by 395
  28. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  29. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  30. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  31. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  32. 2018Video Action Transformer Networkcited by 316
  33. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  34. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  35. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  36. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  37. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  38. 2020Understanding the Difficulty of Training Transformerscited by 210

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 30; 2023: 66; 2024: 86; 2025: 70; 2026: 56.

308 papers.

Sort:

By area:

  1. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  3. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  4. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  5. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  6. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  7. 2023TriDet: Temporal Action Detection with Relative Boundary Modelingcited by 141
  8. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  9. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  10. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  11. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  12. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
38 more
  1. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  2. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  3. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  4. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  5. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
  6. 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
  7. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  8. 2023End-to-End Temporal Action Detection with 1B Parameters Across 1000 Framescited by 47
  9. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  10. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 45
  11. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  12. 2023Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localizationcited by 42
  13. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  14. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  15. 2022Spotting Temporally Precise, Fine-Grained Events in Videocited by 40
  16. 2023MGMAE: Motion Guided Masking for Video Masked Autoencodingcited by 40
  17. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
  18. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  19. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  20. 2022mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensorscited by 37
  21. 2023MammalNet: A Large-scale Video Benchmark for Mammal Recognition and Behavior Understandingcited by 35
  22. 2024360+x: A Panoptic Multi-modal Scene Understanding Datasetcited by 35
  23. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 35
  24. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  25. 2023UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localizationcited by 33
  26. 2024OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understandingcited by 31
  27. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  28. 2023WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognitioncited by 30
  29. 2022BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detectioncited by 28
  30. 2023Action Sensitivity Learning for Temporal Action Localizationcited by 28
  31. 2023TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localizationcited by 27
  32. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  33. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  34. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25
  35. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  36. 2023Self-Feedback DETR for Temporal Action Detectioncited by 22
  37. 2024T-DEED: Temporal-Discriminability Enhancer Encoder-Decoder for Precise Event Spotting in Sports Videoscited by 21
  38. 2023The 7th AI City Challengecited by 20

50 most cited of 308 All 308 (JSON)