mlsanityData to 7 Oct 2026

Long Short-Term Transformer for Online Action Detection

arXiv 2107.03377 ↗First posted 7 Jul 2021cs.CVcited by 87

Not on the map. Why?

Abstract (arXiv)

We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e.g., 2048 frames spanning of up to 8 minutes), together with an LSTR decoder that focuses on a short time window (e.g., 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared to prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS'14, TVSeries, and HACS Segment. Code has been made available at: https://xumingze0308.github.io/projects/lstr

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 55 of its 75 references that are in the map. First posted on arXiv 7 Jul 2021.

Builds on

55 of 75 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  12. 2017Non-local Neural Networkscited by 2,290
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  9. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  12. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  13. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  14. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  15. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  16. 2021Video Transformer Networkcited by 271
  17. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  18. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  19. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  20. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  21. 2017R-C3D: Region Convolutional 3D Network for Temporal Activity Detectioncited by 161
  22. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 147
  23. 2021Towards Long-Form Video Understandingcited by 146
  24. 2021Relaxed Transformer Decoders for Direct Action Proposal Generationcited by 116
  25. 2017TURN TAP: Temporal Unit Regression Network for Temporal Action Proposalscited by 113
  26. 2017CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videoscited by 112
  27. 2021VidTr: Video Transformer Without Convolutionscited by 112
  28. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  29. 2018Temporal Recurrent Networks for Online Action Detectioncited by 79
  30. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  31. 2018Multi-granularity Generator for Temporal Action Proposalcited by 77
  32. 2019Unsupervised Traffic Accident Detection in First-Person Videoscited by 69
  33. 2016Online Action Detectioncited by 68
  34. 2017Learnable pooling with Context Gating for video classificationcited by 58
  35. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48
  36. 2019Learning to Discriminate Information for Online Action Detectioncited by 37
  37. 2019StartNet: Online Detection of Action Start in Untrimmed Videoscited by 21
  38. 2020WOAD: Weakly Supervised Online Action Detection in Untrimmed Videoscited by 19

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 18; 2023: 16; 2024: 17; 2025: 27; 2026: 6.

87 papers.

Sort:

By area:

  1. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  2. 2021PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformercited by 118
  3. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  4. 2022MeMOT: Multi-Object Tracking with Memorycited by 113
  5. 2022TALLFormer: Temporal Action Localization with a Long-memory Transformercited by 78
  6. 2022Video Transformers: A Surveycited by 63
  7. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  8. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  9. 2023PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformercited by 46
  10. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  11. 2022STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognitioncited by 43
  12. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
38 more
  1. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  2. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  3. 2022GateHUB: Gated History Unit with Background Suppression for Online Action Detectioncited by 27
  4. 2022EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithmcited by 27
  5. 2022Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectorscited by 25
  6. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 18
  7. 2023COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformerscited by 17
  8. 2022Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Modelscited by 14
  9. 2022The Wisdom of Crowds: Temporal Progressive Attention for Early Action Predictioncited by 13
  10. 2025Context-Enhanced Memory-Refined Transformer for Online Action Detectioncited by 13
  11. 2022Continual Transformers: Redundancy-Free Attention for Online Inferencecited by 12
  12. 2022Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videoscited by 12
  13. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  14. 2024HERMES: temporal-coHERent long-forM understanding with Episodes and Semanticscited by 11
  15. 2024HAT: History-Augmented Anchor Transformer for Online Temporal Action Localizationcited by 10
  16. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  17. 2023CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformerscited by 9
  18. 2025Action Anticipation from SoccerNet Football Video Broadcastscited by 9
  19. 2023E2E-LOAD: End-to-End Long-form Online Action Detectioncited by 8
  20. 2023Temporal Sentence Grounding in Streaming Videoscited by 8
  21. 2024Online Temporal Action Localization with Memory-Augmented Transformercited by 8
  22. 2024Human Action Anticipation: A Surveycited by 8
  23. 2022An Efficient Framework for Few-shot Skeleton-based Temporal Action Segmentationcited by 7
  24. 2023JOADAA: joint online action detection and action anticipationcited by 7
  25. 2024OnlineTAS: An Online Baseline for Temporal Action Segmentationcited by 7
  26. 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
  27. 2021Learning to Discriminate Information for Online Action Detection: Analysis and Applicationcited by 6
  28. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  29. 2025Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Groundingcited by 5
  30. 2022A Circular Window-based Cascade Transformer for Online Action Detectioncited by 4
  31. 2023Memory-augmented Online Video Anomaly Detectioncited by 4
  32. 2024ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videoscited by 4
  33. 2024Real-Time Human Action Recognition on Embedded Platformscited by 4
  34. 2025Personalized Query Auto-Completion for Long and Short-Term Interests with Adaptive Detoxification Generationcited by 4
  35. 2025Online Generic Event Boundary Detectioncited by 4
  36. 2024AI Foundation Model for Heliophysics: Applications, Design, and Implementationcited by 3
  37. 2025HateClipSeg: A Segment-Level Annotated Dataset for Fine-Grained Hate Video Detectioncited by 3
  38. 2025Track-On2: Enhancing Online Point Tracking with Memorycited by 3

50 most cited of 87 All 87 (JSON)