SAM Audio: Segment Anything in Audio
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.
Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. First posted on arXiv 19 Dec 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2025: 1; 2026: 27.
28 papers.
Sort:
By area:
- 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
- 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
- 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
- 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 4
- 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
- 2026TADA! Tuning Audio Diffusion Models through Activation Steeringcited by 2
- 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 1
- 2026Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Studycited by 1
- 2026JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curationcited by 1
- 2026ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Modelscited by 1
- 2026AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilitiescited by 1
- 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
16 more
- 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
- 2026An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearingcited by 0
- 2026When Audio Separation Hurts Zero-Shot ASR: Evaluating SAM-Audio with Whisper on Bengali and English Speechcited by 0
- 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
- 2026SURF: Separation via Unsupervised Remixing Flowcited by 0
- 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Modelscited by 0
- 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
- 2026A Production-Oriented Framework for Evaluation of SFX Generationcited by 0
- 2026Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Frameworkcited by 0
- 2026Acoustic UAV Detection in Battlefield Scenarios: Handling Noise, Domain Shift, and Weak Labelscited by 0
- 2026TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Filmcited by 0
- 2026Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generationcited by 0
- 2026Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixingcited by 0
- 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0
- 2026SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelcited by 0
- 2026EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streamscited by 0
- 2026EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams8 Oct 2026
- 2026SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model8 Oct 2026
- 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech24 Sep 2026
- 2026Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing24 Sep 2026
- 2026Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation16 Sep 2026
- 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing8 Sep 2026
- 2026TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film30 Aug 2026
- 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation22 Aug 2026
- 2026Acoustic UAV Detection in Battlefield Scenarios: Handling Noise, Domain Shift, and Weak Labels14 Aug 2026
- 2026Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework10 Aug 2026
- 2026AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities17 Jul 2026
- 2026A Production-Oriented Framework for Evaluation of SFX Generation10 Jul 2026
16 more
- 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control28 Jun 2026
- 2026ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models27 Jun 2026
- 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models8 Jun 2026
- 2026SURF: Separation via Unsupervised Remixing Flow3 Jun 2026
- 2026JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation2 Jun 2026
- 2026Cosmos 3: Omnimodal World Models for Physical AI1 Jun 2026
- 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editing18 May 2026
- 2026Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study13 May 2026
- 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing12 Apr 2026
- 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models20 Mar 2026
- 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec6 Mar 2026
- 2026When Audio Separation Hurts Zero-Shot ASR: Evaluating SAM-Audio with Whisper on Bengali and English Speech5 Mar 2026
- 2026An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearing3 Mar 2026
- 2026TADA! Tuning Audio Diffusion Models through Activation Steering12 Feb 2026
- 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks2 Feb 2026
- 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning22 Dec 2025
- 2026TADA! Tuning Audio Diffusion Models through Activation Steeringcited by 2
- 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
- 2026SURF: Separation via Unsupervised Remixing Flowcited by 0
- 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Modelscited by 0
- 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation22 Aug 2026
- 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models8 Jun 2026
- 2026SURF: Separation via Unsupervised Remixing Flow3 Jun 2026
- 2026TADA! Tuning Audio Diffusion Models through Activation Steering12 Feb 2026
- 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
- 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
- 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 4
- 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 1
- 2026Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Studycited by 1
- 2026JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curationcited by 1
- 2026ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Modelscited by 1
- 2026AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilitiescited by 1
- 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
- 2026An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearingcited by 0
- 2026When Audio Separation Hurts Zero-Shot ASR: Evaluating SAM-Audio with Whisper on Bengali and English Speechcited by 0
- 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
12 most cited of 22 All 22 (JSON)
- 2026EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams8 Oct 2026
- 2026SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model8 Oct 2026
- 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech24 Sep 2026
- 2026Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing24 Sep 2026
- 2026Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation16 Sep 2026
- 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing8 Sep 2026
- 2026TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film30 Aug 2026
- 2026Acoustic UAV Detection in Battlefield Scenarios: Handling Noise, Domain Shift, and Weak Labels14 Aug 2026
- 2026Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Framework10 Aug 2026
- 2026AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities17 Jul 2026
- 2026A Production-Oriented Framework for Evaluation of SFX Generation10 Jul 2026
- 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control28 Jun 2026
12 newest of 22 All 22 (JSON)
Similar, not cited
Close in wording, no citation either way
- 2025PromptSep: Generative Audio Separation via Multimodal Prompting
- 2025MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
- 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
- 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
- 2023Audiobox: Unified Audio Generation with Natural Language Prompts
- 2025ZeroSep: Separate Anything in Audio with Zero Training
- 2025U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding