# Positive Sample Propagation along the Audio-Visual Event Line

arXiv: https://arxiv.org/abs/2104.00239 · HTML: https://mlsanity.com/paper/2104.00239/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Dual-modality seq2seq network for audio-visual event localization](https://mlsanity.com/paper/1902.07473/) (2019)
- [Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2008.05789/) (2020)
- [Noise-Aware Unsupervised Deep Lidar-Stereo Fusion](https://mlsanity.com/paper/1904.03868/) (2019)
- [Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data](https://mlsanity.com/paper/2006.01595/) (2020)

## Built on by (most cited first, 30 of 69)

- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning](https://mlsanity.com/paper/2303.11302/) (2023)
- [Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline](https://mlsanity.com/paper/2303.12930/) (2023)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing](https://mlsanity.com/paper/2111.12374/) (2021)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge](https://mlsanity.com/paper/2308.10175/) (2023)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks](https://mlsanity.com/paper/2311.05152/) (2023)
- [Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2204.11573/) (2022)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization](https://mlsanity.com/paper/2210.05060/) (2022)
- [EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer](https://mlsanity.com/paper/2312.04152/) (2023)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Label-anticipated Event Disentanglement for Audio-Visual Video Parsing](https://mlsanity.com/paper/2407.08126/) (2024)
- [Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization](https://mlsanity.com/paper/2106.14118/) (2021)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [Hyperbolic Audio-visual Zero-shot Learning](https://mlsanity.com/paper/2308.12558/) (2023)
- [Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering](https://mlsanity.com/paper/2312.12816/) (2023)
- [Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representations](https://mlsanity.com/paper/2104.08506/) (2021)
- [Improving Audio-Visual Video Parsing with Pseudo Visual Labels](https://mlsanity.com/paper/2303.02344/) (2023)

## Abstract (arXiv)

Visual and audio signals often coexist in natural environments, forming audio-visual events (AVEs). Given a video, we aim to localize video segments containing an AVE and identify its category. In order to learn discriminative features for a classifier, it is pivotal to identify the helpful (or positive) audio-visual segment pairs while filtering out the irrelevant ones, regardless whether they are synchronized or not. To this end, we propose a new positive sample propagation (PSP) module to discover and exploit the closely related audio-visual pairs by evaluating the relationship within every possible pair. It can be done by constructing an all-pair similarity map between each audio and visual segment, and only aggregating the features from the pairs with high similarity scores. To encourage the network to extract high correlated features for positive samples, a new audio-visual pair similarity loss is proposed. We also propose a new weighting branch to better exploit the temporal correlations in weakly supervised setting. We perform extensive experiments on the public AVE dataset and achieve new state-of-the-art accuracy in both fully and weakly supervised settings, thus verifying the effectiveness of our method.