# Unveiling and Mitigating Bias in Audio Visual Segmentation

arXiv: https://arxiv.org/abs/2407.16638 · HTML: https://mlsanity.com/paper/2407.16638/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- [IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding](https://mlsanity.com/paper/2402.18476/) (2024)
- [Overcoming Language Priors in Visual Question Answering with Adversarial Regularization](https://mlsanity.com/paper/1810.03649/) (2018)
- [Self-Critical Reasoning for Robust Visual Question Answering](https://mlsanity.com/paper/1905.09998/) (2019)
- [AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation](https://mlsanity.com/paper/2305.01836/) (2023)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [Self-Supervised Audio-Visual Co-Segmentation](https://mlsanity.com/paper/1904.09013/) (2019)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [Class-aware Sounding Objects Localization via Audiovisual Correspondence](https://mlsanity.com/paper/2112.11749/) (2021)
- [Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics](https://mlsanity.com/paper/2307.16620/) (2023)
- [Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation](https://mlsanity.com/paper/2309.09501/) (2023)
- [Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2407.11820/) (2024)
- [Embodied Visual Active Learning for Semantic Segmentation](https://mlsanity.com/paper/2012.09503/) (2020)
- [Can Textual Semantics Mitigate Sounding Object Segmentation Preference?](https://mlsanity.com/paper/2407.10947/) (2024)
- [Self-Supervised Moving Vehicle Detection from Audio-Visual Cues](https://mlsanity.com/paper/2201.12771/) (2022)
- [Learning content and context with language bias for Visual Question Answering](https://mlsanity.com/paper/2012.11134/) (2020)

## Built on by (most cited first, 11 of 11)

- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization](https://mlsanity.com/paper/2505.11217/) (2025)
- [Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2507.22886/) (2025)
- [Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics](https://mlsanity.com/paper/2503.12840/) (2025)
- [Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment](https://mlsanity.com/paper/2503.12847/) (2025)
- [Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?](https://mlsanity.com/paper/2502.00358/) (2025)
- [Revisiting Audio-Visual Segmentation with Vision-Centric Transformer](https://mlsanity.com/paper/2506.23623/) (2025)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation](https://mlsanity.com/paper/2603.14203/) (2026)
- [Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2608.29121/) (2026)

## Abstract (arXiv)

Community researchers have developed a range of advanced audio-visual segmentation models aimed at improving the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding, which leads to the disregard of important modality information. Generally, the anomalous phenomena are often complex and cannot be directly observed systematically. In this study, we made a pioneering effort with the proper synthetic data to categorize and analyze phenomena as two types "audio priming bias" and "visual prior" according to the source of anomalies. For audio priming bias, to enhance audio sensitivity to different intensities and semantics, a perception module specifically for audio perceives the latent semantic information and incorporates information into a limited set of queries, namely active queries. Moreover, the interaction mechanism related to such active queries in the transformer decoder is customized to adapt to the need for interaction regulating among audio semantics. For visual prior, multiple contrastive training strategies are explored to optimize the model by incorporating a biased branch, without even changing the structure of the model. During experiments, observation demonstrates the presence and the impact that has been produced by the biases of the existing model. Finally, through experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of our methods in handling both types of biases, achieving competitive performance across all three subsets.