mlsanityData to 7 Oct 2026

OadTR: Online Action Detection with Transformers

arXiv 2106.11149 ↗First posted 21 Jun 2021cs.CVcited by 83

Not on the map. Why?

Abstract (arXiv)

Most recent approaches for online action detection tend to apply Recurrent Neural Network (RNN) to capture long-range temporal structure. However, RNN suffers from non-parallelism and gradient vanishing, hence it is hard to be optimized. In this paper, we propose a new encoder-decoder framework based on Transformers, named OadTR, to tackle these problems. The encoder attached with a task token aims to capture the relationships and global interactions between historical observations. The decoder extracts auxiliary information by aggregating anticipated future clip representations. Therefore, OadTR can recognize current actions by encoding historical information and predicting future context simultaneously. We extensively evaluate the proposed OadTR on three challenging datasets: HDD, TVSeries, and THUMOS14. The experimental results show that OadTR achieves higher training and inference speeds than current RNN based approaches, and significantly outperforms the state-of-the-art methods in terms of both mAP and mcAP. Code is available at https://github.com/wangxiang1230/OadTR.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 52 of its 61 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

52 of 61 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2016Layer Normalizationcited by 5,579
  11. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  4. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  7. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  8. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  9. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  10. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  11. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  12. 2020Learning Memory-guided Normality for Anomaly Detectioncited by 278
  13. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  14. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  15. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
  16. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  17. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  18. 2017R-C3D: Region Convolutional 3D Network for Temporal Activity Detectioncited by 161
  19. 2017Single Shot Temporal Action Detectioncited by 123
  20. 2017CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videoscited by 112
  21. 2020End-to-end Lane Shape Prediction with Transformerscited by 111
  22. 2021Temporal Context Aggregation Network for Temporal Action Proposal Refinementcited by 111
  23. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  24. 2018Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoningcited by 102
  25. 2015Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videoscited by 93
  26. 2019Fast Learning of Temporal Action Proposal via Dense Boundary Generatorcited by 89
  27. 2020Boundary Content Graph Neural Network for Temporal Action Proposal Generationcited by 88
  28. 2020Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detectioncited by 83
  29. 2018Temporal Recurrent Networks for Online Action Detectioncited by 79
  30. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  31. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  32. 2016Online Action Detectioncited by 68
  33. 2021Self-Supervised Learning for Semi-Supervised Temporal Action Proposalcited by 39
  34. 2019Learning to Discriminate Information for Online Action Detectioncited by 37
  35. 2020ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesiscited by 34
  36. 2015Joint Inference of Groups, Events and Human Roles in Aerial Videoscited by 30
  37. 2016Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networkscited by 28
  38. 2019StartNet: Online Detection of Action Start in Untrimmed Videoscited by 21

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 19; 2024: 15; 2025: 25; 2026: 7.

83 papers.

Sort:

By area:

  1. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  2. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  3. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  4. 2022Video Transformers: A Surveycited by 63
  5. 2022Flow-Guided Transformer for Video Inpaintingcited by 59
  6. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  7. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  8. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 49
  9. 2022STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognitioncited by 43
  10. 2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
  11. 2023MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognitioncited by 36
  12. 2024PREGO: online mistake detection in PRocedural EGOcentric videoscited by 36
38 more
  1. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  2. 2022Proposal-Free Temporal Action Detection via Global Segmentation Mask Learningcited by 29
  3. 2022GateHUB: Gated History Unit with Background Suppression for Online Action Detectioncited by 27
  4. 2022RCL: Recurrent Continuous Localization for Temporal Action Detectioncited by 25
  5. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  6. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  7. 2022Colar: Effective and Efficient Online Action Detection by Consulting Exemplarscited by 21
  8. 2025Context-Enhanced Memory-Refined Transformer for Online Action Detectioncited by 13
  9. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  10. 2023HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognitioncited by 12
  11. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  12. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  13. 2023InfoGCN++: Learning Representation by Predicting the Future for Online Human Skeleton-based Action Recognitioncited by 11
  14. 2024HAT: History-Augmented Anchor Transformer for Online Temporal Action Localizationcited by 10
  15. 2024TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videoscited by 10
  16. 2023E2E-LOAD: End-to-End Long-form Online Action Detectioncited by 8
  17. 2023Temporal Sentence Grounding in Streaming Videoscited by 8
  18. 2024Online Temporal Action Localization with Memory-Augmented Transformercited by 8
  19. 2024Human Action Anticipation: A Surveycited by 8
  20. 2022SimOn: A Simple Framework for Online Temporal Action Localizationcited by 7
  21. 2023JOADAA: joint online action detection and action anticipationcited by 7
  22. 2023Privacy-preserving Early Detection of Epileptic Seizures in Videoscited by 7
  23. 2024OnlineTAS: An Online Baseline for Temporal Action Segmentationcited by 7
  24. 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
  25. 2023Exploiting Optical Flow Guidance for Transformer-Based Video Inpaintingcited by 6
  26. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  27. 2025Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Groundingcited by 5
  28. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  29. 2022A Circular Window-based Cascade Transformer for Online Action Detectioncited by 4
  30. 2023Learning Road Scene-level Representations via Semantic Region Predictioncited by 4
  31. 2024ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videoscited by 4
  32. 2024Real-Time Human Action Recognition on Embedded Platformscited by 4
  33. 2024Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Contextcited by 4
  34. 2025YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Taskscited by 4
  35. 2025Online Generic Event Boundary Detectioncited by 4
  36. 2022Context-aware Proposal Network for Temporal Action Detectioncited by 3
  37. 2023Enlarging Instance-specific and Class-specific Information for Open-set Action Recognitioncited by 3
  38. 2025HateClipSeg: A Segment-Level Annotated Dataset for Fine-Grained Hate Video Detectioncited by 3

50 most cited of 83 All 83 (JSON)