# Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation

arXiv: https://arxiv.org/abs/2511.22025 · HTML: https://mlsanity.com/paper/2511.22025/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Builds on (most cited first, 30 of 30)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation](https://mlsanity.com/paper/2105.01839/) (2021)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [AVSegFormer: Audio-Visual Segmentation with Transformer](https://mlsanity.com/paper/2307.01146/) (2023)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [ReMamber: Referring Image Segmentation with Mamba Twister](https://mlsanity.com/paper/2403.17839/) (2024)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation](https://mlsanity.com/paper/2304.02970/) (2023)
- [Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes](https://mlsanity.com/paper/2407.10957/) (2024)
- [BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge](https://mlsanity.com/paper/2308.10175/) (2023)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [Audio-Visual Segmentation via Unlabeled Frame Exploitation](https://mlsanity.com/paper/2403.11074/) (2024)
- [Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation](https://mlsanity.com/paper/2312.06462/) (2023)
- [Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation](https://mlsanity.com/paper/2501.08580/) (2025)
- [IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis](https://mlsanity.com/paper/2503.00936/) (2025)
- [Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2407.11820/) (2024)
- [T-VSL: Text-Guided Visual Sound Source Localization in Mixtures](https://mlsanity.com/paper/2404.01751/) (2024)
- [Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge](https://mlsanity.com/paper/2403.17420/) (2024)
- [Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation](https://mlsanity.com/paper/2504.00356/) (2025)
- [CPM: Class-conditional Prompting Machine for Audio-visual Segmentation](https://mlsanity.com/paper/2407.05358/) (2024)
- [Audio-Visual Grounding Referring Expression for Robotic Manipulation](https://mlsanity.com/paper/2109.10571/) (2021)
- [Transavs: End-To-End Audio-Visual Segmentation With Transformer](https://mlsanity.com/paper/2305.07223/) (2023)
- [Multimodal Speech Recognition for Language-Guided Embodied Agents](https://mlsanity.com/paper/2302.14030/) (2023)
- [Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos](https://mlsanity.com/paper/2306.15644/) (2023)
- [Enhancing Sound Source Localization via False Negative Elimination](https://mlsanity.com/paper/2408.16448/) (2024)
- [Progressive Confident Masking Attention Network for Audio-Visual Segmentation](https://mlsanity.com/paper/2406.02345/) (2024)

## Built on by (most cited first, 1 of 1)

- [AeroReformer2: Spoken-Query Referring Segmentation for Aerial Images](https://mlsanity.com/paper/2608.08874/) (2026)

## Abstract (arXiv)

Understanding human instructions is essential for enabling smooth human-robot interaction. In this work, we focus on object grounding, i.e., localizing an object of interest in a visual scene (e.g., an image) based on verbal human instructions. Despite recent progress, a dominant research trend relies on using text as an intermediate representation. These approaches typically transcribe speech to text, extract relevant object keywords, and perform grounding using models pretrained on large text-vision datasets. However, we question both the efficiency and robustness of such transcription-based pipelines. Specifically, we ask: Can we achieve direct audio-visual alignment without relying on text? To explore this possibility, we simplify the task by focusing on grounding from single-word spoken instructions. We introduce a new audio-based grounding dataset that covers a wide variety of objects and diverse human accents. We then adapt and benchmark several models from the closely audio-visual field. Our results demonstrate that direct grounding from audio is not only feasible but, in some cases, even outperforms transcription-based methods, especially in terms of robustness to linguistic variability. Our findings encourage a renewed interest in direct audio grounding and pave the way for more robust and efficient multimodal understanding systems.