mlsanityData to 7 Oct 2026

SAM Audio: Segment Anything in Audio

arXiv 2512.18099 ↗First posted 19 Dec 2025eess.AS cs.CVcited by 28

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

General audio source separation is a key capability for multimodal AI systems that can perceive and reason about sound. Despite substantial progress in recent years, existing separation models are either domain-specific, designed for fixed categories such as speech or music, or limited in controllability, supporting only a single prompting modality such as text. In this work, we present SAM Audio, a foundation model for general audio separation that unifies text, visual, and temporal span prompting within a single framework. Built on a diffusion transformer architecture, SAM Audio is trained with flow matching on large-scale audio data spanning speech, music, and general sounds, and can flexibly separate target sources described by language, visual masks, or temporal spans. The model achieves state-of-the-art performance across a diverse suite of benchmarks, including general sound, speech, music, and musical instrument separation in both in-the-wild and professionally produced audios, substantially outperforming prior general-purpose and specialized systems. Furthermore, we introduce a new real-world separation benchmark with human-labeled multimodal prompts and a reference-free evaluation model that correlates strongly with human judgment.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. First posted on arXiv 19 Dec 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 27.

28 papers.

Sort:

By area:

  1. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  2. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  3. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  4. 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 4
  5. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  6. 2026TADA! Tuning Audio Diffusion Models through Activation Steeringcited by 2
  7. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 1
  8. 2026Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Studycited by 1
  9. 2026JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curationcited by 1
  10. 2026ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Modelscited by 1
  11. 2026AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilitiescited by 1
  12. 2026FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separationcited by 1
16 more
  1. 2026AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editingcited by 1
  2. 2026An Interpretable, Controllable Time-Varying IIR Denoiser for On-Device Assistive Hearingcited by 0
  3. 2026When Audio Separation Hurts Zero-Shot ASR: Evaluating SAM-Audio with Whisper on Bengali and English Speechcited by 0
  4. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  5. 2026SURF: Separation via Unsupervised Remixing Flowcited by 0
  6. 2026Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Modelscited by 0
  7. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  8. 2026A Production-Oriented Framework for Evaluation of SFX Generationcited by 0
  9. 2026Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Frameworkcited by 0
  10. 2026Acoustic UAV Detection in Battlefield Scenarios: Handling Noise, Domain Shift, and Weak Labelscited by 0
  11. 2026TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Filmcited by 0
  12. 2026Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generationcited by 0
  13. 2026Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixingcited by 0
  14. 2026YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speechcited by 0
  15. 2026SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Modelcited by 0
  16. 2026EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streamscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PromptSep: Generative Audio Separation via Multimodal Prompting
  2. 2025MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
  3. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
  4. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
  5. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
  6. 2023Audiobox: Unified Audio Generation with Natural Language Prompts
  7. 2025ZeroSep: Separate Anything in Audio with Zero Training
  8. 2025U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding