# Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization

arXiv: https://arxiv.org/abs/2210.05242 · HTML: https://mlsanity.com/paper/2210.05242/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Boundary Proposal Network for Two-Stage Natural Language Video Localization](https://mlsanity.com/paper/2103.08109/) (2021)
- [Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1608.07017/) (2016)
- [Dual-modality seq2seq network for audio-visual event localization](https://mlsanity.com/paper/1902.07473/) (2019)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Self-supervised Moving Vehicle Tracking with Stereo Sound](https://mlsanity.com/paper/1910.11760/) (2019)
- [Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2008.05789/) (2020)
- [Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation](https://mlsanity.com/paper/2104.02026/) (2021)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes](https://mlsanity.com/paper/2203.13412/) (2022)
- [Unsupervised Sound Localization via Iterative Contrastive Learning](https://mlsanity.com/paper/2104.00315/) (2021)

## Built on by (most cited first, 7 of 7)

- [AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation](https://mlsanity.com/paper/2501.07810/) (2025)
- [Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition](https://mlsanity.com/paper/2406.02554/) (2024)
- [Complementary and Contrastive Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2510.10051/) (2025)
- [CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering](https://mlsanity.com/paper/2405.07451/) (2024)
- [PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling](https://mlsanity.com/paper/2505.23155/) (2025)
- [ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2507.09945/) (2025)
- [EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2605.08723/) (2026)

## Abstract (arXiv)

Audio-visual event (AVE) localization has attracted much attention in recent years. Most existing methods are often limited to independently encoding and classifying each video segment separated from the full video (which can be regarded as the segment-level representations of events). However, they ignore the semantic consistency of the event within the same full video (which can be considered as the video-level representations of events). In contrast to existing methods, we propose a novel video-level semantic consistency guidance network for the AVE localization task. Specifically, we propose an event semantic consistency modeling (ESCM) module to explore video-level semantic information for semantic consistency modeling. It consists of two components: a cross-modal event representation extractor (CERE) and an intra-modal semantic consistency enhancer (ISCE). CERE is proposed to obtain the event semantic information at the video level. Furthermore, ISCE takes video-level event semantics as prior knowledge to guide the model to focus on the semantic continuity of an event within each modality. Moreover, we propose a new negative pair filter loss to encourage the network to filter out the irrelevant segment pairs and a new smooth loss to further increase the gap between different categories of events in the weakly-supervised setting. We perform extensive experiments on the public AVE dataset and outperform the state-of-the-art methods in both fully- and weakly-supervised settings, thus verifying the effectiveness of our method.The code is available at https://github.com/Bravo5542/VSCG.