# Cross-modal Cognitive Consensus guided Audio-Visual Segmentation

arXiv: https://arxiv.org/abs/2310.06259 · HTML: https://mlsanity.com/paper/2310.06259/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2023.

## Built on by (most cited first, 12 of 12)

- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Unsupervised Audio-Visual Segmentation with Modality Alignment](https://mlsanity.com/paper/2403.14203/) (2024)
- [Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.08426/) (2024)
- [From Waveforms to Pixels: A Survey on Audio-Visual Segmentation](https://mlsanity.com/paper/2508.03724/) (2025)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [3D Audio-Visual Segmentation](https://mlsanity.com/paper/2411.02236/) (2024)
- [Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency](https://mlsanity.com/paper/2603.09798/) (2026)
- [Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation](https://mlsanity.com/paper/2509.18912/) (2025)
- [Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation](https://mlsanity.com/paper/2603.14203/) (2026)
- [RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding](https://mlsanity.com/paper/2603.09809/) (2026)
- [Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting](https://mlsanity.com/paper/2607.15890/) (2026)

## Abstract (arXiv)

Audio-Visual Segmentation (AVS) aims to extract the sounding object from a video frame, which is represented by a pixel-wise segmentation mask for application scenarios such as multi-modal video editing, augmented reality, and intelligent robot systems. The pioneering work conducts this task through dense feature-level audio-visual interaction, which ignores the dimension gap between different modalities. More specifically, the audio clip could only provide a Global semantic label in each sequence, but the video frame covers multiple semantic objects across different Local regions, which leads to mislocalization of the representationally similar but semantically different object. In this paper, we propose a Cross-modal Cognitive Consensus guided Network (C3N) to align the audio-visual semantics from the global dimension and progressively inject them into the local regions via an attention mechanism. Firstly, a Cross-modal Cognitive Consensus Inference Module (C3IM) is developed to extract a unified-modal label by integrating audio/visual classification confidence and similarities of modality-agnostic label embeddings. Then, we feed the unified-modal label back to the visual backbone as the explicit semantic-level guidance via a Cognitive Consensus guided Attention Module (CCAM), which highlights the local features corresponding to the interested object. Extensive experiments on the Single Sound Source Segmentation (S4) setting and Multiple Sound Source Segmentation (MS3) setting of the AVSBench dataset demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance. Code is available at https://github.com/ZhaofengSHI/AVS-C3N.