mlsanityData to 7 Oct 2026

Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization

arXiv 2310.03456 ↗First posted 5 Oct 2023cs.CV cs.LG cs.MMcited by 1

Not on the map. Why?

Abstract (arXiv)

Temporal Action Localization (TAL) aims to identify actions' start, end, and class labels in untrimmed videos. While recent advancements using transformer networks and Feature Pyramid Networks (FPN) have enhanced visual feature recognition in TAL tasks, less progress has been made in the integration of audio features into such frameworks. This paper introduces the Multi-Resolution Audio-Visual Feature Fusion (MRAV-FF), an innovative method to merge audio-visual data across different temporal resolutions. Central to our approach is a hierarchical gated cross-attention mechanism, which discerningly weighs the importance of audio information at diverse temporal scales. Such a technique not only refines the precision of regression boundaries but also bolsters classification confidence. Importantly, MRAV-FF is versatile, making it compatible with existing FPN TAL architectures and offering a significant enhancement in performance when audio data is available.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

57 of 69 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  7. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  9. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  10. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  11. 2017Focal Loss for Dense Object Detectioncited by 749
  12. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
38 more
  1. 2019Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selectioncited by 539
  2. 2020Rescaling Egocentric Visioncited by 529
  3. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  4. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  5. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  6. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  7. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  8. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  9. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  10. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  11. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  12. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  13. 2020Temporal Pyramid Network for Action Recognitioncited by 163
  14. 2021Learning Salient Boundary Feature for Anchor-free Temporal Action Localizationcited by 156
  15. 2021End-to-end Temporal Action Detection with Transformercited by 150
  16. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  17. 2019Gaussian Temporal Awareness Networks for Action Localizationcited by 135
  18. 2021Relaxed Transformer Decoders for Direct Action Proposal Generationcited by 116
  19. 2021Temporal Context Aggregation Network for Temporal Action Proposal Refinementcited by 111
  20. 2020Video Self-Stitching Graph Network for Temporal Action Localizationcited by 104
  21. 2019Fast Learning of Temporal Action Proposal via Dense Boundary Generatorcited by 89
  22. 2020Boundary Content Graph Neural Network for Temporal Action Proposal Generationcited by 88
  23. 2022TALLFormer: Temporal Action Localization with a Long-memory Transformercited by 78
  24. 2020TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Taskscited by 77
  25. 2020Revisiting Anchor Mechanisms for Temporal Action Localizationcited by 75
  26. 2021Enriching Local and Global Contexts for Temporal Action Localizationcited by 64
  27. 2020Multi-shot Temporal Event Localization: a Benchmarkcited by 52
  28. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48
  29. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  30. 2021Class Semantics-based Attention for Action Detectioncited by 36
  31. 2021DCAN: Improving Temporal Action Detection via Dual Context Aggregationcited by 36
  32. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
  33. 2022Proposal-Free Temporal Action Detection via Global Segmentation Mask Learningcited by 29
  34. 2022BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detectioncited by 28
  35. 2023TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localizationcited by 27
  36. 2021Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generationcited by 25
  37. 2021Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localizationcited by 21
  38. 2019Scale Matters: Temporal Scale Aggregation Network for Precise Action Localization in Untrimmed Videoscited by 17

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025DEL: Dense Event Localization for Multi-modal Audio-Visual Understandingcited by 1