# SimToken: A Simple Baseline for Referring Audio-Visual Segmentation

arXiv: https://arxiv.org/abs/2509.17537 · HTML: https://mlsanity.com/paper/2509.17537/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [VISA: Reasoning Video Object Segmentation via Large Language Models](https://mlsanity.com/paper/2407.11325/) (2024)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [TAVGBench: Benchmarking Text to Audible-Video Generation](https://mlsanity.com/paper/2404.14381/) (2024)
- [Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes](https://mlsanity.com/paper/2407.10957/) (2024)
- [Towards Open-Vocabulary Audio-Visual Event Localization](https://mlsanity.com/paper/2411.11278/) (2024)
- [Label-anticipated Event Disentanglement for Audio-Visual Video Parsing](https://mlsanity.com/paper/2407.08126/) (2024)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling](https://mlsanity.com/paper/2406.00919/) (2024)
- [Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering](https://mlsanity.com/paper/2312.12816/) (2023)
- [Fine-grained Audible Video Description](https://mlsanity.com/paper/2303.15616/) (2023)
- [Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation](https://mlsanity.com/paper/2503.13068/) (2025)
- [SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes](https://mlsanity.com/paper/2506.01558/) (2025)
- [Patch-level Sounding Object Tracking for Audio-Visual Question Answering](https://mlsanity.com/paper/2412.10749/) (2024)
- [Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing](https://mlsanity.com/paper/2412.11248/) (2024)
- [Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation](https://mlsanity.com/paper/2506.23271/) (2025)
- [Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation](https://mlsanity.com/paper/2508.04418/) (2025)
- [CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization](https://mlsanity.com/paper/2508.04566/) (2025)

## Built on by (most cited first, 6 of 6)

- [Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation](https://mlsanity.com/paper/2602.03892/) (2026)
- [Crab+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation](https://mlsanity.com/paper/2603.04128/) (2026)
- [Face-Guided Sentiment Boundary Enhancement for Weakly-Supervised Temporal Sentiment Localization](https://mlsanity.com/paper/2603.14750/) (2026)
- [Motion-Aware Reasoning from Speech to Mask Tracks: Runner-up Solution for the MeViS-Audio Track of the 8th LSVOS Challenge 2026](https://mlsanity.com/paper/2608.22337/) (2026)
- [Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement](https://mlsanity.com/paper/2609.18282/) (2026)
- [Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation](https://mlsanity.com/paper/2610.12783/) (2026)

## Abstract (arXiv)

Referring Audio-Visual Segmentation (Ref-AVS) aims to segment specific objects in videos based on natural language expressions involving audio, vision, and text information. This task poses significant challenges in cross-modal reasoning and fine-grained object localization. In this paper, we propose a simple framework, SimToken, that integrates a multimodal large language model (MLLM) with the Segment Anything Model (SAM). The MLLM is guided to generate a special semantic token representing the referred object. This compact token, enriched with contextual information from all modalities, acts as a prompt to guide SAM to segment objectsacross video frames. To further improve semantic learning, we introduce a novel target-consistent semantic alignment loss that aligns token embeddings from different expressions but referring to the same object. Experiments on the Ref-AVS benchmark demonstrate that our approach achieves superior performance compared to existing methods.