mlsanityData to 14 Oct 2026

FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec

arXiv 2309.07405 ↗First posted 14 Sep 2023cs.SD cs.AI eess.AScited by 44

Not on the map. Why?

Abstract (arXiv)

This paper presents FunCodec, a fundamental neural speech codec toolkit, which is an extension of the open-source speech processing toolkit FunASR. FunCodec provides reproducible training recipes and inference scripts for the latest neural speech codec models, such as SoundStream and Encodec. Thanks to the unified design with FunASR, FunCodec can be easily integrated into downstream tasks, such as speech recognition. Along with FunCodec, pre-trained models are also provided, which can be used for academic or generalized purposes. Based on the toolkit, we further propose the frequency-domain codec models, FreqCodec, which can achieve comparable speech quality with much lower computation and parameter complexity. Experimental results show that, under the same compression ratio, FunCodec can achieve better reconstruction quality compared with other toolkits and released models. We also demonstrate that the pre-trained models are suitable for downstream tasks, including automatic speech recognition and personalized text-to-speech synthesis. This toolkit is publicly available at https://github.com/alibaba-damo-academy/FunCodec.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

24 of 34 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  9. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  10. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  11. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  12. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
12 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  2. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  3. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
  4. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  5. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  6. 2020SEANet: A Multi-modal Speech Enhancement Networkcited by 37
  7. 2020Real-time Speech Frequency Bandwidth Extensioncited by 22
  8. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  9. 2023LMCodec: A Low Bitrate Speech Codec With Causal Transformer Modelscited by 9
  10. 2022Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Codingcited by 4
  11. 2022High Quality Audio Coding with MDCTNetcited by 4
  12. 2022DRED: Deep REDundancy Coding of Speech Using a Rate-Distortion-Optimized Variational Autoencodercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 14; 2026: 13.

44 papers.

Sort:

By area:

  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  2. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  3. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  4. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  5. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  6. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  7. 2025M-Prometheus: A Suite of Open Multilingual LLM Judgescited by 38
  8. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  9. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  10. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  11. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  12. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
32 more
  1. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  2. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  3. 2026MOSS-TTS Technical Reportcited by 14
  4. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  5. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  6. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  7. 2024Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audiocited by 6
  8. 2025PAST: Phonetic-Acoustic Speech Tokenizercited by 6
  9. 2025LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Modelscited by 5
  10. 2024PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoderscited by 3
  11. 2024FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Ratescited by 3
  12. 2026SAME: A Semantically-Aligned Music Autoencodercited by 3
  13. 2025Analysing the Language of Neural Audio Codecscited by 2
  14. 2026UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstructioncited by 2
  15. 2026OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Datacited by 2
  16. 2024The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challengecited by 1
  17. 2025Autoregressive Speech Enhancement via Acoustic Tokenscited by 1
  18. 2025Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancementcited by 1
  19. 2025Eliminating stability hallucinations in llm-based tts models via attention guidancecited by 1
  20. 2026EuleroDec: A Complex-Valued RVQ-VAE for Efficient and Robust Audio Codingcited by 1
  21. 2026The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecscited by 1
  22. 2026Latent-Mark: An Audio Watermark Robust to Neural Codec Compressioncited by 1
  23. 2026StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Modelcited by 1
  24. 2026Codec-Robust Attacks on Audio LLMscited by 1
  25. 2026Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditionscited by 1
  26. 2025Towards Reliable Audio Deepfake Attribution and Model Recognition: A Multi-Level Autoencoder-Based Frameworkcited by 0
  27. 2025EGGCodec: A Robust Neural Encodec Framework for EGG Reconstruction and F0 Extractioncited by 0
  28. 2025Multilingual Dataset Integration Strategies for Robust Audio Deepfake Detection: A SAFE Challenge Systemcited by 0
  29. 2025Two-Dimensional Quantization for Geometry-Aware Audio Codingcited by 0
  30. 2026A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detectioncited by 0
  31. 2026Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimizationcited by 0
  32. 2026TokenMapper: A Step Toward Interoperable Speech Token Translationcited by 0