mlsanityData to 7 Oct 2026

With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition

arXiv 2111.01024 ↗First posted 1 Nov 2021cs.CV cs.SD eess.AScited by 41

Not on the map. Why?

Abstract (arXiv)

In egocentric videos, actions occur in quick succession. We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance. To incorporate the temporal context, we propose a transformer-based multimodal model that ingests video and audio as input modalities, with an explicit language model providing action sequence context to enhance the predictions. We test our approach on EPIC-KITCHENS and EGTEA datasets reporting state-of-the-art performance. Our ablations showcase the advantage of utilising temporal context as well as incorporating audio input modality and language model to rescore predictions. Code and models at: https://github.com/ekazakos/MTCN.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 60 of its 75 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

60 of 75 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
38 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  4. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  7. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  8. 2015Listen, Attend and Spellcited by 542
  9. 2020Rescaling Egocentric Visioncited by 529
  10. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  11. 2020Multi-modal Transformer for Video Retrievalcited by 395
  12. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  13. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  14. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  15. 2017Temporal Relational Reasoning in Videoscited by 324
  16. 2018Video Action Transformer Networkcited by 316
  17. 2021Video Transformer Networkcited by 271
  18. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  19. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  20. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  21. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  22. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  23. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  24. 2015Character-Aware Neural Language Modelscited by 129
  25. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  26. 2020Multi-Modal Domain Adaptation for Fine-Grained Action Recognitioncited by 115
  27. 2020Multi-modal Dense Video Captioningcited by 113
  28. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  29. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  30. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  31. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  32. 2021Space-time Mixing Attention for Video Transformercited by 84
  33. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  34. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  35. 2018Object Level Visual Reasoning in Videoscited by 67
  36. 2018LSTA: Long Short-Term Attention for Egocentric Action Recognitioncited by 57
  37. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  38. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 10; 2024: 8; 2025: 4; 2026: 2.

41 papers.

Sort:

By area:

  1. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  2. 2022Learning Video Representations from Large Language Modelscited by 204
  3. 2023An Outlook into the Future of Egocentric Visioncited by 84
  4. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  5. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  6. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  7. 2021MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detectioncited by 42
  8. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  9. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  10. 2023Ego-Only: Egocentric Action Detection without Exocentric Transferringcited by 30
  11. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  12. 2022Vision Transformers for Action Recognition: A Surveycited by 25
29 more
  1. 2022EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queriescited by 25
  2. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  3. 2022M&M Mix: A Multimodal Multiview Transformer Ensemblecited by 19
  4. 2024TIM: A Time Interval Machine for Audio-Visual Action Recognitioncited by 15
  5. 2023The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspectivecited by 14
  6. 2024Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognitioncited by 14
  7. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  8. 2022Continuous Human Action Recognition for Human-Machine Interaction: A Reviewcited by 10
  9. 2022SOS! Self-supervised Learning Over Sets Of Handled Objects In Egocentric Action Recognitioncited by 8
  10. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8
  11. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  12. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  13. 2025EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrainingcited by 5
  14. 2022OWL (Observe, Watch, Listen): Audiovisual Temporal Context for Localizing Actions in Egocentric Videoscited by 4
  15. 2022ConTra: (Con)text (Tra)nsformer for Cross-Modal Video Retrievalcited by 3
  16. 2023EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge: Mixed Sequences Predictioncited by 3
  17. 2023Centre Stage: Centricity-based Audio-Visual Temporal Action Detectioncited by 3
  18. 2022Students taught by multimodal teachers are superior action recognizerscited by 2
  19. 2024Test-Time Adaptation for Combating Missing Modalities in Egocentric Videoscited by 2
  20. 2022Bringing Online Egocentric Action Recognition into the wildcited by 1
  21. 2023Co-Occurrence Matters: Learning Action Relation for Temporal Action Localizationcited by 1
  22. 2023Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localizationcited by 1
  23. 2024FILS: Self-Supervised Video Feature Prediction In Semantic Language Spacecited by 1
  24. 2025Domain Generalization using Action Sequences for Egocentric Action Recognitioncited by 1
  25. 2022A Survey on Human Action Recognitioncited by 0
  26. 2025Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalitiescited by 0
  27. 2025Conceptualizing Multi-scale Wavelet Attention and Ray-based Encoding for Human-Object Interaction Detectioncited by 0
  28. 2026Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learningcited by 0
  29. 2026G3Ego: Gaze-Guided Graphs for Egocentric Action Understandingcited by 0