# BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis

arXiv: https://arxiv.org/abs/2205.14807 · HTML: https://mlsanity.com/paper/2205.14807/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 30 May 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2108.02938/) (2021)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents](https://mlsanity.com/paper/2201.00308/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [D2C: Diffusion-Denoising Models for Few-shot Conditional Generation](https://mlsanity.com/paper/2106.06819/) (2021)
- [Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation](https://mlsanity.com/paper/2007.09902/) (2020)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Visually Informed Binaural Audio Generation without Binaural Audios](https://mlsanity.com/paper/2104.06162/) (2021)
- [Building and Evaluation of a Real Room Impulse Response Dataset](https://mlsanity.com/paper/1811.06795/) (2018)
- [Bilateral Denoising Diffusion Models](https://mlsanity.com/paper/2108.11514/) (2021)
- [InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Training](https://mlsanity.com/paper/2202.03751/) (2022)

## Built on by (most cited first, 30 of 44)

- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation](https://mlsanity.com/paper/2306.17115/) (2023)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound](https://mlsanity.com/paper/2405.00233/) (2024)
- [Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis](https://mlsanity.com/paper/2312.03491/) (2023)
- [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models](https://mlsanity.com/paper/2303.13472/) (2023)
- [SGDiff: A Style Guided Diffusion Model for Fashion Synthesis](https://mlsanity.com/paper/2308.07605/) (2023)
- [OmniAudio: Generating Spatial Audio from 360-Degree Video](https://mlsanity.com/paper/2504.14906/) (2025)
- [The Age of Synthetic Realities: Challenges and Opportunities](https://mlsanity.com/paper/2306.11503/) (2023)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [I4VGen: Image as Free Stepping Stone for Text-to-Video Generation](https://mlsanity.com/paper/2406.02230/) (2024)
- [DiffSketching: Sketch Control Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2305.18812/) (2023)
- [Expensive Multi-Objective Bayesian Optimization Based on Diffusion Models](https://mlsanity.com/paper/2405.08674/) (2024)
- [ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2301.12935/) (2023)
- [DiffAnt: Diffusion Models for Action Anticipation](https://mlsanity.com/paper/2311.15991/) (2023)
- [Structured Diffusion Models with Mixture of Gaussians as Prior Distribution](https://mlsanity.com/paper/2410.19149/) (2024)
- [Improving Diffusion Models for ECG Imputation with an Augmented Template Prior](https://mlsanity.com/paper/2310.15742/) (2023)
- [BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models](https://mlsanity.com/paper/2505.22865/) (2025)
- [DiffSED: Sound Event Detection with Denoising Diffusion](https://mlsanity.com/paper/2308.07293/) (2023)
- [FIRST: A Million-Entry Dataset for Text-Driven Fashion Synthesis and Design](https://mlsanity.com/paper/2311.07414/) (2023)
- [Neural Fourier Shift for Binaural Speech Rendering](https://mlsanity.com/paper/2211.00878/) (2022)
- [R2-Diff: Denoising by diffusion as a refinement of retrieved motion for image-based motion prediction](https://mlsanity.com/paper/2306.09483/) (2023)
- [DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin](https://mlsanity.com/paper/2309.00883/) (2023)
- [DualSpec: Text-to-spatial-audio Generation via Dual-Spectrogram Guided Diffusion Model](https://mlsanity.com/paper/2502.18952/) (2025)

## Abstract (arXiv)

Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This synthesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtrations, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into a common part that shared by the left and right channels as well as a specific part that differs in each channel. Accordingly, we propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Specifically, in the first stage, the common information of the binaural audio is generated with a single-channel diffusion model conditioned on the mono audio, based on which the binaural audio is generated by a two-channel diffusion model in the second stage. Combining this novel perspective of two-stage synthesis with advanced generative models (i.e., the diffusion models),the proposed BinauralGrad is able to generate accurate and high-fidelity binaural audio samples. Experiment results show that on a benchmark dataset, BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics (Wave L2: 0.128 vs. 0.157, MOS: 3.80 vs. 3.61). The generated audio samples (https://speechresearch.github.io/binauralgrad) and code (https://github.com/microsoft/NeuralSpeech/tree/master/BinauralGrad) are available online.