mlsanityData to 7 Oct 2026

EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition

arXiv 1908.08498 ↗First posted 22 Aug 2019cs.CVcited by 226

Not on the map. Why?

Abstract (arXiv)

We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities -- RGB, Flow and Audio -- and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality and fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as well as late-fusion of modalities. We demonstrate the importance of audio in egocentric vision, on per-class basis, for identifying actions as well as interacting objects. Our method achieves state of the art results on both the seen and unseen test sets of the largest egocentric dataset: EPIC-Kitchens, on all metrics using the public leaderboard.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 42; 2022: 52; 2023: 57; 2024: 41; 2025: 26; 2026: 8.

226 papers.

Sort:

By area:

  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  5. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  6. 2022Egocentric Video-Language Pretrainingcited by 231
  7. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  8. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 136
  9. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  10. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 98
  11. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  12. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
38 more
  1. 2021Space-time Mixing Attention for Video Transformercited by 84
  2. 2023An Outlook into the Future of Egocentric Visioncited by 84
  3. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  4. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  5. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  6. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62
  7. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  8. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  9. 2022MAViL: Masked Audio-Video Learnerscited by 56
  10. 2021E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognitioncited by 50
  11. 2023SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalizationcited by 48
  12. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  13. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  14. 2023Learning Procedure-aware Video Representation from Instructional Videos and Their Narrationscited by 46
  15. 2021MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognitioncited by 44
  16. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  17. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  18. 2023Annotation-free Audio-Visual Segmentationcited by 44
  19. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  20. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  21. 2024GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformercited by 42
  22. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  23. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  24. 2021Learning to Anticipate Egocentric Actions by Imaginationcited by 40
  25. 2023Egocentric Audio-Visual Object Localizationcited by 40
  26. 2023A Light Weight Model for Active Speaker Detectioncited by 39
  27. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  28. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  29. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  30. 2022MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domaincited by 36
  31. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  32. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  33. 2021Multiview Pseudo-Labeling for Semi-supervised Learning from Videocited by 32
  34. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  35. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32
  36. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  37. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  38. 2022Egocentric Prediction of Action Target in 3Dcited by 31

50 most cited of 226 All 226 (JSON)