# Neural Fourier Shift for Binaural Speech Rendering

arXiv: https://arxiv.org/abs/2211.00878 · HTML: https://mlsanity.com/paper/2211.00878/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Fast Training of Convolutional Networks through FFTs](https://mlsanity.com/paper/1312.5851/) (2013)
- [BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis](https://mlsanity.com/paper/2205.14807/) (2022)
- [Deep Impulse Responses: Estimating and Parameterizing Filters with Deep Networks](https://mlsanity.com/paper/2202.03416/) (2022)
- [Differentiable Artificial Reverberation](https://mlsanity.com/paper/2105.13940/) (2021)
- [End-to-End Binaural Speech Synthesis](https://mlsanity.com/paper/2207.03697/) (2022)
- [Global HRTF Interpolation via Learned Affine Transformation of Hyper-conditioned Features](https://mlsanity.com/paper/2204.02637/) (2022)

## Built on by (most cited first, 3 of 3)

- [Zero-Shot Mono-to-Binaural Speech Synthesis](https://mlsanity.com/paper/2412.08356/) (2024)
- [TTMBA: Towards Text To Multiple Sources Binaural Audio Generation](https://mlsanity.com/paper/2507.16564/) (2025)
- [Gaussian Process Regression of Steering Vectors With Physics-Aware Deep Composite Kernels for Augmented Listening](https://mlsanity.com/paper/2509.02571/) (2025)

## Abstract (arXiv)

We present a neural network for rendering binaural speech from given monaural audio, position, and orientation of the source. Most of the previous works have focused on synthesizing binaural speeches by conditioning the positions and orientations in the feature space of convolutional neural networks. These synthesis approaches are powerful in estimating the target binaural speeches even for in-the-wild data but are difficult to generalize for rendering the audio from out-of-distribution domains. To alleviate this, we propose Neural Fourier Shift (NFS), a novel network architecture that enables binaural speech rendering in the Fourier space. Specifically, utilizing a geometric time delay based on the distance between the source and the receiver, NFS is trained to predict the delays and scales of various early reflections. NFS is efficient in both memory and computational cost, is interpretable, and operates independently of the source domain by its design. Experimental results show that NFS performs comparable to the previous studies on the benchmark dataset, even with its 25 times lighter memory and 6 times fewer calculations.