# Revisiting Audio-Visual Segmentation with Vision-Centric Transformer

arXiv: https://arxiv.org/abs/2506.23623 · HTML: https://mlsanity.com/paper/2506.23623/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 44 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [AVSegFormer: Audio-Visual Segmentation with Transformer](https://mlsanity.com/paper/2307.01146/) (2023)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer](https://mlsanity.com/paper/2309.07929/) (2023)
- [Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation](https://mlsanity.com/paper/2304.02970/) (2023)
- [Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics](https://mlsanity.com/paper/2307.16620/) (2023)

## Built on by (most cited first, 6 of 6)

- [Learning What To Hear: Boosting Sound-Source Association For Robust Audiovisual Instance Segmentation](https://mlsanity.com/paper/2509.22740/) (2025)
- [Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation](https://mlsanity.com/paper/2512.20117/) (2025)
- [Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation](https://mlsanity.com/paper/2603.08967/) (2026)
- [Technical Report for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Pretraining-Diverse Ensemble of Foundation Vision Encoders for Robust Outdoor Scene Understanding](https://mlsanity.com/paper/2606.23113/) (2026)
- [Audio-Visual Segmentation via Depth-Guided Collaborative Modeling](https://mlsanity.com/paper/2608.16285/) (2026)
- [SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention](https://mlsanity.com/paper/2610.15764/) (2026)

## Abstract (arXiv)

Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where object queries are derived from audio features. However, audio-centric Transformers suffer from two limitations: perception ambiguity caused by the mixed nature of audio, and weakened dense prediction ability due to visual detail loss. To address these limitations, we propose a new Vision-Centric Transformer (VCT) framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information, enabling queries to better distinguish between different sounding objects from mixed audio and accurately delineate their contours. Additionally, we also introduce a Prototype Prompted Query Generation (PPQG) module within our VCT framework to generate vision-derived queries that are both semantically aware and visually rich through audio prototype prompting and pixel context grouping, facilitating audio-visual information aggregation. Extensive experiments demonstrate that our VCT framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset. The code is available at https://github.com/spyflying/VCT_AVS.