mlsanityData to 14 Oct 2026

End-to-end Learning of Action Detection from Frame Glimpses in Videos

arXiv 1511.06984 ↗First posted 22 Nov 2015cs.CV cs.LGcited by 101

Not on the map. Why?

Abstract (arXiv)

In this work we introduce a fully end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions. Our intuition is that the process of detecting actions is naturally one of observation and refinement: observing moments in video, and refining hypotheses about when an action is occurring. Based on this insight, we formulate our model as a recurrent neural network-based agent that interacts with a video over time. The agent observes video frames and decides both where to look next and when to emit a prediction. Since backpropagation is not adequate in this non-differentiable setting, we use REINFORCE to learn the agent's decision policy. Our model achieves state-of-the-art results on the THUMOS'14 and ActivityNet datasets while observing only a fraction (2% or less) of the video frames.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 45; 2022: 21; 2023: 16; 2024: 12; 2025: 6; 2026: 1.

101 papers.

Sort:

By area:

  1. 2021Dynamic Neural Networks: A Surveycited by 408
  2. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  3. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  4. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  5. 2021End-to-end Temporal Action Detection with Transformercited by 151
  6. 2022FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessmentcited by 107
  7. 2022Temporal Alignment Networks for Long-term Videocited by 84
  8. 2021Adaptive Focus for Efficient Video Recognitioncited by 77
  9. 2021Group-aware Contrastive Regression for Action Quality Assessmentcited by 75
  10. 2021Object-Region Video Transformerscited by 73
  11. 2021Enriching Local and Global Contexts for Temporal Action Localizationcited by 64
  12. 2021FineAction: A Fine-Grained Video Dataset for Temporal Action Localizationcited by 61
38 more
  1. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  2. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 59
  3. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  4. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  5. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48
  6. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  7. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 48
  8. 2022An Empirical Study of End-to-End Temporal Action Detectioncited by 40
  9. 2021The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methodscited by 38
  10. 2021AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognitioncited by 37
  11. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  12. 2021Dynamic Network Quantization for Efficient Video Inferencecited by 36
  13. 2021DCAN: Improving Temporal Action Detection via Dual Context Aggregationcited by 36
  14. 2022Probabilistic Representations for Video Contrastive Learningcited by 34
  15. 2021Modeling Multi-Label Action Dependencies for Temporal Action Localizationcited by 32
  16. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  17. 2021A Survey on Temporal Sentence Grounding in Videoscited by 29
  18. 2021Graph Convolutional Module for Temporal Action Localization in Videoscited by 29
  19. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 29
  20. 2021Coarse-Fine Networks for Temporal Activity Detection in Videoscited by 28
  21. 2021Action Unit Memory Network for Weakly Supervised Temporal Action Localizationcited by 26
  22. 2021Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activitiescited by 26
  23. 2021Foreground-Action Consistency Network for Weakly Supervised Temporal Action Localizationcited by 26
  24. 2021Variable-rate discrete representation learningcited by 24
  25. 2021No frame left behind: Full Video Action Recognitioncited by 24
  26. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  27. 2022OCSampler: Compressing Videos to One Clip with Single-step Samplingcited by 23
  28. 2022OpenTAL: Towards Open Set Temporal Action Localizationcited by 23
  29. 2023Self-Feedback DETR for Temporal Action Detectioncited by 22
  30. 2021Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localizationcited by 21
  31. 2021Shifted Chunk Transformer for Spatio-Temporal Representational Learningcited by 20
  32. 2021Reinforced Attention for Few-Shot Learning and Beyondcited by 18
  33. 2022AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognitioncited by 18
  34. 2023SpotEM: Efficient Video Search for Episodic Memorycited by 17
  35. 2021SVIP: Sequence VerIfication for Procedures in Videoscited by 16
  36. 2022An Efficient Spatio-Temporal Pyramid Transformer for Action Detectioncited by 16
  37. 2021VA-RED2: Video Adaptive Redundancy Reductioncited by 15
  38. 2022NSNet: Non-saliency Suppression Sampler for Efficient Video Recognitioncited by 15

50 most cited of 101 All 101 (JSON)