mlsanityData to 14 Oct 2026

Temporal Relational Reasoning in Videos

arXiv 1711.08496 ↗First posted 22 Nov 2017cs.CVcited by 324

Not on the map. Why?

Abstract (arXiv)

Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.

Key facts. Cited by 324 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 122; 2022: 75; 2023: 50; 2024: 34; 2025: 25; 2026: 18.

324 papers.

Sort:

By area:

  1. 2021ViViT: A Video Vision Transformercited by 1,620
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  3. 2021Multiscale Vision Transformerscited by 897
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  5. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  6. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  7. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  8. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  9. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  10. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  11. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  12. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
38 more
  1. 2022Frozen CLIP Models are Efficient Video Learnerscited by 182
  2. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  3. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  4. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  5. 2021Towards Long-Form Video Understandingcited by 146
  6. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  7. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  8. 2021VidTr: Video Transformer Without Convolutionscited by 112
  9. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  10. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  11. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  12. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 93
  13. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 91
  14. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  15. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  16. 2021Temporal-Relational CrossTransformers for Few-Shot Action Recognitioncited by 84
  17. 2021Space-time Mixing Attention for Video Transformercited by 84
  18. 2021Adaptive Focus for Efficient Video Recognitioncited by 77
  19. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  20. 2021Object-Region Video Transformerscited by 73
  21. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  22. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  23. 2021ACTION-Net: Multipath Excitation for Action Recognitioncited by 63
  24. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  25. 2021FrameExit: Conditional Early Exiting for Efficient Video Recognitioncited by 59
  26. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  27. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  28. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  29. 2023ViTs for SITS: Vision Transformers for Satellite Image Time Seriescited by 56
  30. 2021Semi-Supervised Action Recognition with Temporal Contrastive Learningcited by 50
  31. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  32. 2022Hybrid Relation Guided Set Matching for Few-shot Action Recognitioncited by 50
  33. 2021MGSampler: An Explainable Sampling Strategy for Video Action Recognitioncited by 49
  34. 2021Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixingcited by 49
  35. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 49
  36. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  37. 2021Learning Cross-modal Contrastive Features for Video Domain Adaptationcited by 47
  38. 2021MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognitioncited by 46

50 most cited of 324 All 324 (JSON)