mlsanityData to 14 Oct 2026

End-to-End Binaural Speech Synthesis

arXiv 2207.03697 ↗First posted 8 Jul 2022cs.SD cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we present an end-to-end binaural speech synthesis system that combines a low-bitrate audio codec with a powerful binaural decoder that is capable of accurate speech binauralization while faithfully reconstructing environmental factors like ambient noise or reverb. The network is a modified vector-quantized variational autoencoder, trained with several carefully designed objectives, including an adversarial loss. We evaluate the proposed system on an internal binaural dataset with objective metrics and a perceptual study. Results show that the proposed approach matches the ground truth data more closely than previous methods. In particular, we demonstrate the capability of the adversarial loss in capturing environment effects needed to create an authentic auditory scene.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 8 Jul 2022.

Builds on

20 of 33 references.

References · 20

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  3. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  8. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  9. 2018cGANs with Projection Discriminatorcited by 254
  10. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  11. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  12. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
8 more
  1. 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 49
  2. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  3. 2017Wavenet based low rate speech codingcited by 33
  4. 2022Deep Impulse Responses: Estimating and Parameterizing Filters with Deep Networkscited by 20
  5. 2021Generative Speech Coding with Predictive Variance Regularizationcited by 18
  6. 2019Self-supervised Audio Spatialization with Correspondence Classifiercited by 10
  7. 2020Efficient And Scalable Neural Residual Waveform Coding With Collaborative Quantizationcited by 7
  8. 2021DPLM: A Deep Perceptual Spatial-Audio Localization Metriccited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  2. 2024Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Modelscited by 6
  3. 2022Neural Fourier Shift for Binaural Speech Renderingcited by 3
  4. 2024Zero-Shot Mono-to-Binaural Speech Synthesiscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
  2. 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
  3. 2021HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding
  4. 2025Learning to Upsample and Upmix Audio in the Latent Domain
  5. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders
  6. 2026BinauralVAE: Spatial Audio Reconstruction For World Models
  7. 2025Re-Bottleneck: Latent Re-Structuring for Neural Audio Autoencoders
  8. 2026AudioTQ: A Data-Oblivious 6-Bit CPU Audio Codec via Randomized Hadamard Rotation and Lloyd-Max Quantization