# End-to-End Neural Speech Coding for Real-Time Communications

arXiv: https://arxiv.org/abs/2201.09429 · HTML: https://mlsanity.com/paper/2201.09429/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 17 of 17)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)
- [Wavenet based low rate speech coding](https://mlsanity.com/paper/1712.01120/) (2017)
- [ICASSP 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2009.06122/) (2020)
- [Generative Speech Coding with Predictive Variance Regularization](https://mlsanity.com/paper/2102.09660/) (2021)
- [Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding](https://mlsanity.com/paper/1906.07769/) (2019)
- [High-quality speech coding with SampleRNN](https://mlsanity.com/paper/1811.03021/) (2018)
- [Interactive Speech and Noise Modeling for Speech Enhancement](https://mlsanity.com/paper/2012.09408/) (2020)
- [Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm](https://mlsanity.com/paper/2010.10727/) (2020)
- [Improving Opus Low Bit Rate Quality with Neural Speech Synthesis](https://mlsanity.com/paper/1905.04628/) (2019)
- [Source coding of audio signals with a generative model](https://mlsanity.com/paper/2001.09847/) (2020)

## Built on by (most cited first, 7 of 7)

- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming](https://mlsanity.com/paper/2211.08428/) (2022)
- [Disentangled Feature Learning for Real-Time Neural Speech Coding](https://mlsanity.com/paper/2211.11960/) (2022)
- [Latent-Domain Predictive Neural Speech Coding](https://mlsanity.com/paper/2207.08363/) (2022)
- [Cross-Scale Vector Quantization for Scalable Neural Speech Coding](https://mlsanity.com/paper/2207.03067/) (2022)
- [FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Rates](https://mlsanity.com/paper/2409.17635/) (2024)
- [AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes](https://mlsanity.com/paper/2308.07221/) (2023)

## Abstract (arXiv)

Deep-learning based methods have shown their advantages in audio coding over traditional ones but limited attention has been paid on real-time communications (RTC). This paper proposes the TFNet, an end-to-end neural speech codec with low latency for RTC. It takes an encoder-temporal filtering-decoder paradigm that has seldom been investigated in audio coding. An interleaved structure is proposed for temporal filtering to capture both short-term and long-term temporal dependencies. Furthermore, with end-to-end optimization, the TFNet is jointly optimized with speech enhancement and packet loss concealment, yielding a one-for-all network for three tasks. Both subjective and objective results demonstrate the efficiency of the proposed TFNet.