mlsanityData to 14 Oct 2026

Context-Enhanced Memory-Refined Transformer for Online Action Detection

arXiv 2503.18359 ↗First posted 24 Mar 2025cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Online Action Detection (OAD) detects actions in streaming videos using past observations. State-of-the-art OAD approaches model past observations and their interactions with an anticipated future. The past is encoded using short- and long-term memories to capture immediate and long-range dependencies, while anticipation compensates for missing future context. We identify a training-inference discrepancy in existing OAD methods that hinders learning effectiveness. The training uses varying lengths of short-term memory, while inference relies on a full-length short-term memory. As a remedy, we propose a Context-enhanced Memory-Refined Transformer (CMeRT). CMeRT introduces a context-enhanced encoder to improve frame representations using additional near-past context. It also features a memory-refined decoder to leverage near-future generation to enhance performance. CMeRT achieves state-of-the-art in online detection and anticipation on THUMOS'14, CrossTask, and EPIC-Kitchens-100.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 49 of its 55 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

49 of 55 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  9. 2021ViViT: A Video Vision Transformercited by 1,620
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
37 more
  1. 2020Rescaling Egocentric Visioncited by 540
  2. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  3. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  4. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  5. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  6. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  7. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  8. 2021ASFormer: Transformer for Action Segmentationcited by 168
  9. 2021End-to-end Temporal Action Detection with Transformercited by 151
  10. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 120
  11. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  12. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  13. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  14. 2021Long Short-Term Transformer for Online Action Detectioncited by 88
  15. 2020Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detectioncited by 83
  16. 2021OadTR: Online Action Detection with Transformerscited by 83
  17. 2018Temporal Recurrent Networks for Online Action Detectioncited by 80
  18. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 78
  19. 2019Grounding Human-to-Vehicle Advice for Self-driving Vehiclescited by 71
  20. 2016Online Action Detectioncited by 69
  21. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  22. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  23. 2019Learning to Discriminate Information for Online Action Detectioncited by 37
  24. 2015Joint Inference of Groups, Events and Human Roles in Aerial Videoscited by 30
  25. 2016Online Human Action Detection using Joint Classification-Regression Recurrent Neural Networkscited by 28
  26. 2022GateHUB: Gated History Unit with Background Suppression for Online Action Detectioncited by 27
  27. 2022C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentationcited by 23
  28. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  29. 2019StartNet: Online Detection of Action Start in Untrimmed Videoscited by 21
  30. 2020WOAD: Weakly Supervised Online Action Detection in Untrimmed Videoscited by 19
  31. 2018Online Detection of Action Start in Untrimmed, Streaming Videoscited by 17
  32. 2021Temporally smooth online action detection using cycle-consistent future anticipationcited by 13
  33. 2021Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actionscited by 10
  34. 2023JOADAA: joint online action detection and action anticipationcited by 7
  35. 2021Learning to Discriminate Information for Online Action Detection: Analysis and Applicationcited by 6
  36. 2024X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalizationcited by 6
  37. 2024Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustmentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understandingcited by 19
  2. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4
  3. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  4. 2026Efficiency Follows Global-Local Decouplingcited by 1
  5. 2026iPay: Integrated Payment Action Recognition via Multimodal Networks and Adaptive Spatial Prior Learningcited by 1
  6. 2026ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistencycited by 1
  7. 2025Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understandingcited by 0
  8. 2025From Detection to Anticipation: Online Understanding of Struggles across Various Tasks and Activitiescited by 0
  9. 2026Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detectioncited by 0
  10. 2026Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detectioncited by 0
  11. 2026OZ-TAL: Online Zero-Shot Temporal Action Localizationcited by 0
  12. 2026MA-DLE: Speech-based Automatic Depression Level Estimation via Memory Augmentationcited by 0
1 more
  1. 2026Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videoscited by 0