# LINK: Adaptive Modality Interaction for Audio-Visual Video Parsing

arXiv: https://arxiv.org/abs/2412.20872 · HTML: https://mlsanity.com/paper/2412.20872/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing](https://mlsanity.com/paper/2111.12374/) (2021)
- [Language-driven All-in-one Adverse Weather Removal](https://mlsanity.com/paper/2312.01381/) (2023)
- [Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks](https://mlsanity.com/paper/2311.05152/) (2023)
- [Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2204.11573/) (2022)
- [Label-anticipated Event Disentanglement for Audio-Visual Video Parsing](https://mlsanity.com/paper/2407.08126/) (2024)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2405.10690/) (2024)
- [Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization](https://mlsanity.com/paper/2211.02077/) (2022)
- [CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing](https://mlsanity.com/paper/2310.07517/) (2023)

## Built on by (most cited first, 3 of 3)

- [MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing](https://mlsanity.com/paper/2507.01384/) (2025)
- [TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph](https://mlsanity.com/paper/2509.04086/) (2025)
- [EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2605.08723/) (2026)

## Abstract (arXiv)

Audio-visual video parsing focuses on classifying videos through weak labels while identifying events as either visible, audible, or both, alongside their respective temporal boundaries. Many methods ignore that different modalities often lack alignment, thereby introducing extra noise during modal interaction. In this work, we introduce a Learning Interaction method for Non-aligned Knowledge (LINK), designed to equilibrate the contributions of distinct modalities by dynamically adjusting their input during event prediction. Additionally, we leverage the semantic information of pseudo-labels as a priori knowledge to mitigate noise from other modalities. Our experimental findings demonstrate that our model outperforms existing methods on the LLP dataset.