# SAVE: Segment Audio-Visual Easy way using Segment Anything Model

arXiv: https://arxiv.org/abs/2407.02004 · HTML: https://mlsanity.com/paper/2407.02004/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 35)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2206.06522/) (2022)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Explicit Visual Prompting for Low-Level Structure Segmentations](https://mlsanity.com/paper/2303.10883/) (2023)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [AVSegFormer: Audio-Visual Segmentation with Transformer](https://mlsanity.com/paper/2307.01146/) (2023)
- [Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction](https://mlsanity.com/paper/2112.13528/) (2021)
- [Making a Case for 3D Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2008.11516/) (2020)
- [AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation](https://mlsanity.com/paper/2305.01836/) (2023)
- [Parameter Efficient Multimodal Transformers for Video Representation Learning](https://mlsanity.com/paper/2012.04124/) (2020)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer](https://mlsanity.com/paper/2309.07929/) (2023)
- [Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation](https://mlsanity.com/paper/2307.13236/) (2023)
- [Class-aware Sounding Objects Localization via Audiovisual Correspondence](https://mlsanity.com/paper/2112.11749/) (2021)
- [Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics](https://mlsanity.com/paper/2307.16620/) (2023)
- [Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes](https://mlsanity.com/paper/2203.13412/) (2022)
- [Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks](https://mlsanity.com/paper/2311.05152/) (2023)
- [Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation](https://mlsanity.com/paper/2309.09501/) (2023)

## Built on by (most cited first, 4 of 4)

- [How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?](https://mlsanity.com/paper/2601.08133/) (2026)
- [Audio Visual Segmentation Through Text Embeddings](https://mlsanity.com/paper/2502.16359/) (2025)
- [VAInpaint: Zero-Shot Video-Audio inpainting framework with LLMs-driven Module](https://mlsanity.com/paper/2509.17022/) (2025)
- [SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts](https://mlsanity.com/paper/2603.22732/) (2026)

## Abstract (arXiv)

The primary aim of Audio-Visual Segmentation (AVS) is to precisely identify and locate auditory elements within visual scenes by accurately predicting segmentation masks at the pixel level. Achieving this involves comprehensively considering data and model aspects to address this task effectively. This study presents a lightweight approach, SAVE, which efficiently adapts the pre-trained segment anything model (SAM) to the AVS task. By incorporating an image encoder adapter into the transformer blocks to better capture the distinct dataset information and proposing a residual audio encoder adapter to encode the audio features as a sparse prompt, our proposed model achieves effective audio-visual fusion and interaction during the encoding stage. Our proposed method accelerates the training and inference speed by reducing the input resolution from 1024 to 256 pixels while achieving higher performance compared with the previous SOTA. Extensive experimentation validates our approach, demonstrating that our proposed model outperforms other SOTA methods significantly. Moreover, leveraging the pre-trained model on synthetic data enhances performance on real AVSBench data, achieving 84.59 mIoU on the S4 (V1S) subset and 70.28 mIoU on the MS3 (V1M) set with only 256 pixels for input images. This increases up to 86.16 mIoU on the S4 (V1S) and 70.83 mIoU on the MS3 (V1M) with inputs of 1024 pixels.