# Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation

arXiv: https://arxiv.org/abs/2304.02970 · HTML: https://mlsanity.com/paper/2304.02970/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 66 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)

## Built on by (most cited first, 30 of 34)

- [AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation](https://mlsanity.com/paper/2412.15191/) (2024)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization](https://mlsanity.com/paper/2505.11217/) (2025)
- [SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes](https://mlsanity.com/paper/2506.01558/) (2025)
- [Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation](https://mlsanity.com/paper/2506.23271/) (2025)
- [Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics](https://mlsanity.com/paper/2503.12840/) (2025)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment](https://mlsanity.com/paper/2503.12847/) (2025)
- [AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection](https://mlsanity.com/paper/2504.04495/) (2025)
- [AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation](https://mlsanity.com/paper/2408.01708/) (2024)
- [Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?](https://mlsanity.com/paper/2502.00358/) (2025)
- [Revisiting Audio-Visual Segmentation with Vision-Centric Transformer](https://mlsanity.com/paper/2506.23623/) (2025)
- [Leveraging Foundation models for Unsupervised Audio-Visual Segmentation](https://mlsanity.com/paper/2309.06728/) (2023)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation](https://mlsanity.com/paper/2501.02786/) (2025)
- [From Waveforms to Pixels: A Survey on Audio-Visual Segmentation](https://mlsanity.com/paper/2508.03724/) (2025)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [3D Audio-Visual Segmentation](https://mlsanity.com/paper/2411.02236/) (2024)
- [AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting](https://mlsanity.com/paper/2506.01015/) (2025)
- [SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation](https://mlsanity.com/paper/2508.00782/) (2025)
- [MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learning](https://mlsanity.com/paper/2512.00115/) (2025)
- [IDPro: Flexible Interactive Video Object Segmentation by ID-queried Concurrent Propagation](https://mlsanity.com/paper/2401.12480/) (2024)
- [OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models](https://mlsanity.com/paper/2505.01448/) (2025)
- [Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization](https://mlsanity.com/paper/2505.05343/) (2025)
- [Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models](https://mlsanity.com/paper/2506.06537/) (2025)
- [Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation](https://mlsanity.com/paper/2511.22025/) (2025)
- [SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2603.01431/) (2026)
- [Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation](https://mlsanity.com/paper/2603.08967/) (2026)
- [Learning Visual Affordance from Audio](https://mlsanity.com/paper/2512.02005/) (2025)

## Abstract (arXiv)

Audio-visual segmentation (AVS) is a challenging task that involves accurately segmenting sounding objects based on audio-visual cues. The effectiveness of audio-visual learning critically depends on achieving accurate cross-modal alignment between sound and visual objects. Successful audio-visual learning requires two essential components: 1) a challenging dataset with high-quality pixel-level multi-class annotated images associated with audio files, and 2) a model that can establish strong links between audio information and its corresponding visual object. However, these requirements are only partially addressed by current methods, with training sets containing biased audio-visual data, and models that generalise poorly beyond this biased training set. In this work, we propose a new cost-effective strategy to build challenging and relatively unbiased high-quality audio-visual segmentation benchmarks. We also propose a new informative sample mining method for audio-visual supervised contrastive learning to leverage discriminative contrastive samples to enforce cross-modal understanding. We show empirical results that demonstrate the effectiveness of our benchmark. Furthermore, experiments conducted on existing AVS datasets and on our new benchmark show that our method achieves state-of-the-art (SOTA) segmentation accuracy.