# Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

arXiv: https://arxiv.org/abs/2303.12930 · HTML: https://mlsanity.com/paper/2303.12930/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [End-to-end Temporal Action Detection with Transformer](https://mlsanity.com/paper/2106.10271/) (2021)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Video Self-Stitching Graph Network for Temporal Action Localization](https://mlsanity.com/paper/2011.14598/) (2020)
- [Boundary Content Graph Neural Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/2008.01432/) (2020)
- [Revisiting Anchor Mechanisms for Temporal Action Localization](https://mlsanity.com/paper/2008.09837/) (2020)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)

## Built on by (most cited first, 30 of 46)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos](https://mlsanity.com/paper/2411.19772/) (2024)
- [MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World](https://mlsanity.com/paper/2401.08577/) (2024)
- [Audio-centric Video Understanding Benchmark without Text Shortcut](https://mlsanity.com/paper/2503.19951/) (2025)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding](https://mlsanity.com/paper/2403.16276/) (2024)
- [AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs](https://mlsanity.com/paper/2506.05328/) (2025)
- [Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies](https://mlsanity.com/paper/2410.21170/) (2024)
- [LoVA: Long-form Video-to-Audio Generation](https://mlsanity.com/paper/2409.15157/) (2024)
- [MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks](https://mlsanity.com/paper/2506.07016/) (2025)
- [Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization](https://mlsanity.com/paper/2409.07967/) (2024)
- [CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2405.10690/) (2024)
- [CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization](https://mlsanity.com/paper/2408.01952/) (2024)
- [CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2508.04566/) (2025)
- [ChronusOmni: Improving Time Awareness of Omni Large Language Models](https://mlsanity.com/paper/2512.09841/) (2025)
- [Reframe Anything: LLM Agent for Open World Video Reframing](https://mlsanity.com/paper/2403.06070/) (2024)
- [UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization](https://mlsanity.com/paper/2404.03179/) (2024)
- [Language-based Audio Moment Retrieval](https://mlsanity.com/paper/2409.15672/) (2024)
- [OMCAT: Omni Context Aware Transformer](https://mlsanity.com/paper/2410.12109/) (2024)
- [PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling](https://mlsanity.com/paper/2505.23155/) (2025)
- [Audio-visual Event Localization on Portrait Mode Short Videos](https://mlsanity.com/paper/2504.06884/) (2025)
- [Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness](https://mlsanity.com/paper/2504.16936/) (2025)
- [AudioStory: Generating Long-Form Narrative Audio with Large Language Models](https://mlsanity.com/paper/2508.20088/) (2025)
- [CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries](https://mlsanity.com/paper/2511.15131/) (2025)
- [Hear What Matters! Text-conditioned Selective Video-to-Audio Generation](https://mlsanity.com/paper/2512.02650/) (2025)
- [UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2505.09615/) (2025)
- [VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.21556/) (2025)
- [CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.09266/) (2025)
- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models](https://mlsanity.com/paper/2602.20981/) (2026)

## Abstract (arXiv)

Existing audio-visual event localization (AVE) handles manually trimmed videos with only a single instance in each of them. However, this setting is unrealistic as natural videos often contain numerous audio-visual events with different categories. To better adapt to real-life applications, in this paper we focus on the task of dense-localizing audio-visual events, which aims to jointly localize and recognize all audio-visual events occurring in an untrimmed video. The problem is challenging as it requires fine-grained audio-visual scene and context understanding. To tackle this problem, we introduce the first Untrimmed Audio-Visual (UnAV-100) dataset, which contains 10K untrimmed videos with over 30K audio-visual events. Each video has 2.8 audio-visual events on average, and the events are usually related to each other and might co-occur as in real-life scenes. Next, we formulate the task using a new learning-based framework, which is capable of fully integrating audio and visual modalities to localize audio-visual events with various lengths and capture dependencies between them in a single pass. Extensive experiments demonstrate the effectiveness of our method as well as the significance of multi-scale cross-modal perception and dependency modeling for this task.