# SUNAC: Source-aware Unified Neural Audio Codec

arXiv: https://arxiv.org/abs/2511.16126 · HTML: https://mlsanity.com/paper/2511.16126/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 34 of its 36 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models](https://mlsanity.com/paper/2308.16692/) (2023)
- [ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric](https://mlsanity.com/paper/2004.09584/) (2020)
- [UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation](https://mlsanity.com/paper/2106.07889/) (2021)
- [Discrete Audio Tokens: More Than a Survey!](https://mlsanity.com/paper/2506.10274/) (2025)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model](https://mlsanity.com/paper/2505.15670/) (2025)
- [The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks](https://mlsanity.com/paper/2110.09958/) (2021)
- [TF-Locoformer: Transformer with Local Modeling by Convolution for Speech Separation and Enhancement](https://mlsanity.com/paper/2408.03440/) (2024)
- [Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning](https://mlsanity.com/paper/2405.18386/) (2024)
- [Learning Source Disentanglement in Neural Audio Codec](https://mlsanity.com/paper/2409.11228/) (2024)
- [Source-Aware Neural Speech Coding for Noisy Speech Compression](https://mlsanity.com/paper/2008.12889/) (2020)
- [Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders](https://mlsanity.com/paper/2102.06610/) (2021)
- [GASS: Generalizing Audio Source Separation with Large-scale Data](https://mlsanity.com/paper/2310.00140/) (2023)
- [Exploring Text-Queried Sound Event Detection with Audio Source Separation](https://mlsanity.com/paper/2409.13292/) (2024)
- [Hyperbolic Audio Source Separation](https://mlsanity.com/paper/2212.05008/) (2022)
- [U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding](https://mlsanity.com/paper/2505.13880/) (2025)
- [Towards a Japanese Full-duplex Spoken Dialogue System](https://mlsanity.com/paper/2506.02979/) (2025)
- [An End-to-End Speech Summarization Using Large Language Model](https://mlsanity.com/paper/2407.02005/) (2024)
- [Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation](https://mlsanity.com/paper/2408.00205/) (2024)

## Abstract (arXiv)

Neural audio codecs (NACs) provide compact representations that can be leveraged in many downstream applications, in particular large language models. Yet most NACs encode mixtures of multiple sources in an entangled manner, which may impede efficient downstream processing in applications that need access to only a subset of the sources (e.g., analysis of a particular type of sound, transcription of a given speaker, etc). To address this, we propose a source-aware codec that encodes individual sources directly from mixtures, conditioned on source type prompts. This enables user-driven selection of which source(s) to encode, including separately encoding multiple sources of the same type (e.g., multiple speech signals). Experiments show that our model achieves competitive resynthesis and separation quality relative to a cascade of source separation followed by a conventional NAC, with lower computational cost.