# Contrastive Positive Sample Propagation along the Audio-Visual Event Line

arXiv: https://arxiv.org/abs/2211.09980 · HTML: https://mlsanity.com/paper/2211.09980/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 80 references that are in the map.
- First posted on arXiv 18 Nov 2022.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Contrastive Learning for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/2006.09920/) (2020)

## Built on by (most cited first, 30 of 50)

- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [TAVGBench: Benchmarking Text to Audible-Video Generation](https://mlsanity.com/paper/2404.14381/) (2024)
- [Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics](https://mlsanity.com/paper/2307.16620/) (2023)
- [BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge](https://mlsanity.com/paper/2308.10175/) (2023)
- [Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture](https://mlsanity.com/paper/2403.07347/) (2024)
- [EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer](https://mlsanity.com/paper/2312.04152/) (2023)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Label-anticipated Event Disentanglement for Audio-Visual Video Parsing](https://mlsanity.com/paper/2407.08126/) (2024)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering](https://mlsanity.com/paper/2312.12816/) (2023)
- [Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimization](https://mlsanity.com/paper/2507.15765/) (2025)
- [Improving Audio-Visual Video Parsing with Pseudo Visual Labels](https://mlsanity.com/paper/2303.02344/) (2023)
- [TIM: A Time Interval Machine for Audio-Visual Action Recognition](https://mlsanity.com/paper/2404.05559/) (2024)
- [Boosting Audio Visual Question Answering via Key Semantic-Aware Cues](https://mlsanity.com/paper/2407.20693/) (2024)
- [Patch-level Sounding Object Tracking for Audio-Visual Question Answering](https://mlsanity.com/paper/2412.10749/) (2024)
- [Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing](https://mlsanity.com/paper/2412.11248/) (2024)
- [Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization](https://mlsanity.com/paper/2409.07967/) (2024)
- [Exploiting Ensemble Learning for Cross-View Isolated Sign Language Recognition](https://mlsanity.com/paper/2502.02196/) (2025)
- [Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation](https://mlsanity.com/paper/2506.23271/) (2025)
- [Unsupervised Audio-Visual Segmentation with Modality Alignment](https://mlsanity.com/paper/2403.14203/) (2024)
- [CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2508.04566/) (2025)
- [SimToken: A Simple Baseline for Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2509.17537/) (2025)
- [Leveraging Foundation models for Unsupervised Audio-Visual Segmentation](https://mlsanity.com/paper/2309.06728/) (2023)
- [Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition](https://mlsanity.com/paper/2406.02554/) (2024)
- [Semantic Residual for Multimodal Unified Discrete Representation](https://mlsanity.com/paper/2412.19128/) (2024)
- [MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2507.07015/) (2025)
- [UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization](https://mlsanity.com/paper/2404.03179/) (2024)
- [Open-set Cross Modal Generalization via Multimodal Unified Representation](https://mlsanity.com/paper/2507.14935/) (2025)

## Abstract (arXiv)

Visual and audio signals often coexist in natural environments, forming audio-visual events (AVEs). Given a video, we aim to localize video segments containing an AVE and identify its category. It is pivotal to learn the discriminative features for each video segment. Unlike existing work focusing on audio-visual feature fusion, in this paper, we propose a new contrastive positive sample propagation (CPSP) method for better deep feature representation learning. The contribution of CPSP is to introduce the available full or weak label as a prior that constructs the exact positive-negative samples for contrastive learning. Specifically, the CPSP involves comprehensive contrastive constraints: pair-level positive sample propagation (PSP), segment-level and video-level positive sample activation (PSAS and PSAV). Three new contrastive objectives are proposed (i.e., Lavpsp, Lspsa, and Lvpsa) and introduced into both the fully and weakly supervised AVE localization. To draw a complete picture of the contrastive learning in AVE localization, we also study the self-supervised positive sample propagation (SSPSP). As a result, CPSP is more helpful to obtain the refined audio-visual features that are distinguishable from the negatives, thus benefiting the classifier prediction. Extensive experiments on the AVE and the newly collected VGGSound-AVEL100k datasets verify the effectiveness and generalization ability of our method.