# Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization

arXiv: https://arxiv.org/abs/2106.14118 · HTML: https://mlsanity.com/paper/2106.14118/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 27 Jun 2021.

## Builds on (most cited first, 30 of 39)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals](https://mlsanity.com/paper/1703.06189/) (2017)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [Fast Learning of Temporal Action Proposal via Dense Boundary Generator](https://mlsanity.com/paper/1911.04127/) (2019)
- [Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1608.07017/) (2016)
- [Multi-granularity Generator for Temporal Action Proposal](https://mlsanity.com/paper/1811.11524/) (2018)
- [TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks](https://mlsanity.com/paper/2011.11479/) (2020)
- [Speaker Diarization with LSTM](https://mlsanity.com/paper/1710.10468/) (2017)
- [Diagnosing Error in Temporal Action Detectors](https://mlsanity.com/paper/1807.10706/) (2018)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1712.07271/) (2017)
- [BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation](https://mlsanity.com/paper/2009.07641/) (2020)
- [CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016](https://mlsanity.com/paper/1608.00797/) (2016)

## Built on by (most cited first, 21 of 21)

- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [Epic-Sounds: A Large-scale Dataset of Actions That Sound](https://mlsanity.com/paper/2302.00646/) (2023)
- [Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization](https://mlsanity.com/paper/2305.01979/) (2023)
- [UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2308.14395/) (2023)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization](https://mlsanity.com/paper/2411.10193/) (2024)
- [Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization](https://mlsanity.com/paper/2408.01532/) (2024)
- [OWL (Observe, Watch, Listen): Audiovisual Temporal Context for Localizing Actions in Egocentric Videos](https://mlsanity.com/paper/2202.04947/) (2022)
- [UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization](https://mlsanity.com/paper/2404.03179/) (2024)
- [AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization](https://mlsanity.com/paper/2511.18993/) (2025)
- [Enhancing Temporal Action Localization: Advanced S6 Modeling with Recurrent Mechanism](https://mlsanity.com/paper/2407.13078/) (2024)
- [Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization](https://mlsanity.com/paper/2506.08493/) (2025)
- [Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval](https://mlsanity.com/paper/2508.04273/) (2025)
- [Next-Frame Feature Prediction for Multimodal Deepfake Detection and Temporal Localization](https://mlsanity.com/paper/2511.10212/) (2025)
- [Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization](https://mlsanity.com/paper/2310.03456/) (2023)
- [DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding](https://mlsanity.com/paper/2506.23196/) (2025)
- [ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detection](https://mlsanity.com/paper/2508.17282/) (2025)
- [Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization](https://mlsanity.com/paper/2204.06228/) (2022)
- [LoCC: Detection and Localization of Lip-Syncing Deepfakes via Counterfactual Frame Consistency](https://mlsanity.com/paper/2606.22772/) (2026)
- [MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization](https://mlsanity.com/paper/2607.00902/) (2026)
- [UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization](https://mlsanity.com/paper/2607.04498/) (2026)

## Abstract (arXiv)

State of the art architectures for untrimmed video Temporal Action Localization (TAL) have only considered RGB and Flow modalities, leaving the information-rich audio modality totally unexploited. Audio fusion has been explored for the related but arguably easier problem of trimmed (clip-level) action recognition. However, TAL poses a unique set of challenges. In this paper, we propose simple but effective fusion-based approaches for TAL. To the best of our knowledge, our work is the first to jointly consider audio and video modalities for supervised TAL. We experimentally show that our schemes consistently improve performance for state of the art video-only TAL approaches. Specifically, they help achieve new state of the art performance on large-scale benchmark datasets - ActivityNet-1.3 (54.34 mAP@0.5) and THUMOS14 (57.18 mAP@0.5). Our experiments include ablations involving multiple fusion schemes, modality combinations and TAL architectures. Our code, models and associated data are available at https://github.com/skelemoa/tal-hmo.