mlsanityData to 19 Oct 2026

SUNAC: Source-aware Unified Neural Audio Codec

arXiv 2511.16126 ↗First posted 20 Nov 2025eess.AS eess.SPcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Neural audio codecs (NACs) provide compact representations that can be leveraged in many downstream applications, in particular large language models. Yet most NACs encode mixtures of multiple sources in an entangled manner, which may impede efficient downstream processing in applications that need access to only a subset of the sources (e.g., analysis of a particular type of sound, transcription of a given speaker, etc). To address this, we propose a source-aware codec that encodes individual sources directly from mixtures, conditioned on source type prompts. This enables user-driven selection of which source(s) to encode, including separately encoding multiple sources of the same type (e.g., multiple speech signals). Experiments show that our model achieves competitive resynthesis and separation quality relative to a cascade of source separation followed by a conventional NAC, with lower computational cost.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 34 of its 36 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

34 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners11 Sep
  2. 2025FasTUSS: Faster Task-Aware Unified Source Separation15 Jul
  3. 2024Task-Aware Unified Source Separation31 Oct
  4. 2024Learning Source Disentanglement in Neural Audio Codec17 Sep
  5. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models31 Aug
  6. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 28

  1. 2017Attention Is All You Needcited by 61,180
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,812
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,420
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  5. 2022High Fidelity Neural Audio Compressioncited by 917
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 711
  7. 2018SDR - half-baked or well done?cited by 626
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  9. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 281
  10. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 138
  11. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 120
  12. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 107
16 more
  1. 2025Discrete Audio Tokens: More Than a Survey!cited by 53
  2. 2024DASB - Discrete Audio and Speech Benchmarkcited by 48
  3. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 40
  4. 2021The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtrackscited by 24
  5. 2024TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancementcited by 20
  6. 2024Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuningcited by 18
  7. 2020Source-Aware Neural Speech Coding for Noisy Speech Compressioncited by 12
  8. 2021Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoderscited by 12
  9. 2023GASS: Generalizing Audio Source Separation with Large-scale Datacited by 11
  10. 2024Exploring Text-Queried Sound Event Detection with Audio Source Separationcited by 9
  11. 2022Hyperbolic Audio Source Separationcited by 8
  12. 2025U-SAM: An audio language Model for Unified Speech, Audio, and Music Understandingcited by 7
  13. 2025Towards a Japanese Full-duplex Spoken Dialogue Systemcited by 7
  14. 2024An End-to-End Speech Summarization Using Large Language Modelcited by 5
  15. 2024Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillationcited by 5
  16. 2025Towards Bitrate-Efficient and Noise-Robust Speech Coding with Variable Bitrate RVQcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents
  2. 2024Towards Audio Codec-based Speech Separation
  3. 2026Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks
  4. 2025UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
  5. 2025Towards Evaluating Generative Audio: Insights from Neural Audio Codec Embedding Distances
  6. 2025Towards Neural Audio Codec Source Parsing
  7. 2024HILCodec: High-Fidelity and Lightweight Neural Audio Codec
  8. 2026The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs