# Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation

arXiv: https://arxiv.org/abs/2312.06462 · HTML: https://mlsanity.com/paper/2312.06462/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 47 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation](https://mlsanity.com/paper/2101.08833/) (2021)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)

## Built on by (most cited first, 26 of 26)

- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [CPM: Class-conditional Prompting Machine for Audio-visual Segmentation](https://mlsanity.com/paper/2407.05358/) (2024)
- [Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics](https://mlsanity.com/paper/2503.12840/) (2025)
- [Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment](https://mlsanity.com/paper/2503.12847/) (2025)
- [AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation](https://mlsanity.com/paper/2408.01708/) (2024)
- [Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?](https://mlsanity.com/paper/2502.00358/) (2025)
- [Revisiting Audio-Visual Segmentation with Vision-Centric Transformer](https://mlsanity.com/paper/2506.23623/) (2025)
- [Implicit Counterfactual Learning for Audio-Visual Segmentation](https://mlsanity.com/paper/2507.20740/) (2025)
- [From Waveforms to Pixels: A Survey on Audio-Visual Segmentation](https://mlsanity.com/paper/2508.03724/) (2025)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [3D Audio-Visual Segmentation](https://mlsanity.com/paper/2411.02236/) (2024)
- [UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2505.09615/) (2025)
- [AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting](https://mlsanity.com/paper/2506.01015/) (2025)
- [How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?](https://mlsanity.com/paper/2601.08133/) (2026)
- [IDPro: Flexible Interactive Video Object Segmentation by ID-queried Concurrent Propagation](https://mlsanity.com/paper/2401.12480/) (2024)
- [Collaborative Hybrid Propagator for Temporal Misalignment in Audio-Visual Segmentation](https://mlsanity.com/paper/2412.08161/) (2024)
- [Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation](https://mlsanity.com/paper/2509.18912/) (2025)
- [Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation](https://mlsanity.com/paper/2511.22025/) (2025)
- [SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2603.01431/) (2026)
- [Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation](https://mlsanity.com/paper/2603.14203/) (2026)
- [Learning Visual Affordance from Audio](https://mlsanity.com/paper/2512.02005/) (2025)
- [AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes](https://mlsanity.com/paper/2606.02724/) (2026)
- [AeroReformer2: Spoken-Query Referring Segmentation for Aerial Images](https://mlsanity.com/paper/2608.08874/) (2026)
- [Audio-Visual Segmentation via Depth-Guided Collaborative Modeling](https://mlsanity.com/paper/2608.16285/) (2026)
- [Binaural Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2609.32180/) (2026)
- [SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention](https://mlsanity.com/paper/2610.15764/) (2026)

## Abstract (arXiv)

Recently, an audio-visual segmentation (AVS) task has been introduced, aiming to group pixels with sounding objects within a given video. This task necessitates a first-ever audio-driven pixel-level understanding of the scene, posing significant challenges. In this paper, we propose an innovative audio-visual transformer framework, termed COMBO, an acronym for COoperation of Multi-order Bilateral relatiOns. For the first time, our framework explores three types of bilateral entanglements within AVS: pixel entanglement, modality entanglement, and temporal entanglement. Regarding pixel entanglement, we employ a Siam-Encoder Module (SEM) that leverages prior knowledge to generate more precise visual features from the foundational model. For modality entanglement, we design a Bilateral-Fusion Module (BFM), enabling COMBO to align corresponding visual and auditory signals bi-directionally. As for temporal entanglement, we introduce an innovative adaptive inter-frame consistency loss according to the inherent rules of temporal. Comprehensive experiments and ablation studies on AVSBench-object (84.7 mIoU on S4, 59.2 mIou on MS3) and AVSBench-semantic (42.1 mIoU on AVSS) datasets demonstrate that COMBO surpasses previous state-of-the-art methods. Code and more results will be publicly available at https://yannqi.github.io/AVS-COMBO/.