# Zero-Shot Mono-to-Binaural Speech Synthesis

arXiv: https://arxiv.org/abs/2412.08356 · HTML: https://mlsanity.com/paper/2412.08356/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis](https://mlsanity.com/paper/2205.14807/) (2022)
- [SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping](https://mlsanity.com/paper/2203.16749/) (2022)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)
- [Deep Impulse Responses: Estimating and Parameterizing Filters with Deep Networks](https://mlsanity.com/paper/2202.03416/) (2022)
- [WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration](https://mlsanity.com/paper/2210.01029/) (2022)
- [Self-Supervised Visual Acoustic Matching](https://mlsanity.com/paper/2307.15064/) (2023)
- [Differentiable Artificial Reverberation](https://mlsanity.com/paper/2105.13940/) (2021)
- [Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio](https://mlsanity.com/paper/2311.06285/) (2023)
- [End-to-End Binaural Speech Synthesis](https://mlsanity.com/paper/2207.03697/) (2022)
- [Neural Fourier Shift for Binaural Speech Rendering](https://mlsanity.com/paper/2211.00878/) (2022)
- [DopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect](https://mlsanity.com/paper/2212.07000/) (2022)

## Built on by (most cited first, 1 of 1)

- [Physics-Aware Deepfake Detection via Distance-Speech Consistency](https://mlsanity.com/paper/2610.18485/) (2026)

## Abstract (arXiv)

We present ZeroBAS, a neural method to synthesize binaural audio from monaural audio recordings and positional information without training on any binaural data. To our knowledge, this is the first published zero-shot neural approach to mono-to-binaural audio synthesis. Specifically, we show that a parameter-free geometric time warping and amplitude scaling based on source location suffices to get an initial binaural synthesis that can be refined by iteratively applying a pretrained denoising vocoder. Furthermore, we find this leads to generalization across room conditions, which we measure by introducing a new dataset, TUT Mono-to-Binaural, to evaluate state-of-the-art monaural-to-binaural synthesis methods on unseen conditions. Our zero-shot method is perceptually on-par with the performance of supervised methods on the standard mono-to-binaural dataset, and even surpasses them on our out-of-distribution TUT Mono-to-Binaural dataset. Our results highlight the potential of pretrained generative audio models and zero-shot learning to unlock robust binaural audio synthesis.