# Binaural Audio Generation via Multi-task Learning

arXiv: https://arxiv.org/abs/2109.00748 · HTML: https://mlsanity.com/paper/2109.00748/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2021.

## Built on by (most cited first, 5 of 5)

- [AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis](https://mlsanity.com/paper/2406.08920/) (2024)
- [Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation](https://mlsanity.com/paper/2210.16472/) (2022)
- [An Overview of Color Transfer and Style Transfer for Images and Videos](https://mlsanity.com/paper/2204.13339/) (2022)
- [CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation](https://mlsanity.com/paper/2501.02786/) (2025)
- [MTLComb: multi-task learning combining regression and classification tasks for joint feature selection](https://mlsanity.com/paper/2405.09886/) (2024)

## Abstract (arXiv)

We present a learning-based approach for generating binaural audio from mono audio using multi-task learning. Our formulation leverages additional information from two related tasks: the binaural audio generation task and the flipped audio classification task. Our learning model extracts spatialization features from the visual and audio input, predicts the left and right audio channels, and judges whether the left and right channels are flipped. First, we extract visual features using ResNet from the video frames. Next, we perform binaural audio generation and flipped audio classification using separate subnetworks based on visual features. Our learning method optimizes the overall loss based on the weighted sum of the losses of the two tasks. We train and evaluate our model on the FAIR-Play dataset and the YouTube-ASMR dataset. We perform quantitative and qualitative evaluations to demonstrate the benefits of our approach over prior techniques.