# Revisit Weakly-Supervised Audio-Visual Video Parsing from the Language Perspective

arXiv: https://arxiv.org/abs/2306.00595 · HTML: https://mlsanity.com/paper/2306.00595/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Learning Visual Representations with Caption Annotations](https://mlsanity.com/paper/2008.01392/) (2020)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Learning Visual N-Grams from Web Data](https://mlsanity.com/paper/1612.09161/) (2016)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [Dual-modality seq2seq network for audio-visual event localization](https://mlsanity.com/paper/1902.07473/) (2019)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)

## Built on by (most cited first, 14 of 14)

- [Can CLIP Help Sound Source Localization?](https://mlsanity.com/paper/2311.04066/) (2023)
- [Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing](https://mlsanity.com/paper/2412.11248/) (2024)
- [CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2508.04566/) (2025)
- [PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling](https://mlsanity.com/paper/2505.23155/) (2025)
- [MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing](https://mlsanity.com/paper/2507.01384/) (2025)
- [Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Models](https://mlsanity.com/paper/2507.18858/) (2025)
- [UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2505.09615/) (2025)
- [Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization](https://mlsanity.com/paper/2505.05343/) (2025)
- [Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learning](https://mlsanity.com/paper/2507.21588/) (2025)
- [TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph](https://mlsanity.com/paper/2509.04086/) (2025)
- [A Decade of Deep Learning: A Survey on The Magnificent Seven](https://mlsanity.com/paper/2412.16188/) (2024)
- [SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts](https://mlsanity.com/paper/2603.22732/) (2026)
- [EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2605.08723/) (2026)
- [SCoPE: Training-Free Audio-Visual Event Perception via Sparse Cross-Modal Prior Exchange](https://mlsanity.com/paper/2608.07923/) (2026)

## Abstract (arXiv)

We focus on the weakly-supervised audio-visual video parsing task (AVVP), which aims to identify and locate all the events in audio/visual modalities. Previous works only concentrate on video-level overall label denoising across modalities, but overlook the segment-level label noise, where adjacent video segments (i.e., 1-second video clips) may contain different events. However, recognizing events in the segment is challenging because its label could be any combination of events that occur in the video. To address this issue, we consider tackling AVVP from the language perspective, since language could freely describe how various events appear in each segment beyond fixed labels. Specifically, we design language prompts to describe all cases of event appearance for each video. Then, the similarity between language prompts and segments is calculated, where the event of the most similar prompt is regarded as the segment-level label. In addition, to deal with the mislabeled segments, we propose to perform dynamic re-weighting on the unreliable segments to adjust their labels. Experiments show that our simple yet effective approach outperforms state-of-the-art methods by a large margin.