# CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation

arXiv: https://arxiv.org/abs/2501.02786 · HTML: https://mlsanity.com/paper/2501.02786/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 6 Jan 2025.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [A Survey on Multi-view Learning](https://mlsanity.com/paper/1304.5634/) (2013)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [A Survey of Multi-View Representation Learning](https://mlsanity.com/paper/1610.01206/) (2016)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [Semantic Image Synthesis via Adversarial Learning](https://mlsanity.com/paper/1707.06873/) (2017)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation](https://mlsanity.com/paper/2007.09902/) (2020)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [Pooling Methods in Deep Neural Networks, a Review](https://mlsanity.com/paper/2009.07485/) (2020)
- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [Visually Informed Binaural Audio Generation without Binaural Audios](https://mlsanity.com/paper/2104.06162/) (2021)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation](https://mlsanity.com/paper/2304.02970/) (2023)
- [Understanding Test-Time Augmentation](https://mlsanity.com/paper/2402.06892/) (2024)

## Built on by (most cited first, 4 of 4)

- [ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation](https://mlsanity.com/paper/2512.03036/) (2025)
- [AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting](https://mlsanity.com/paper/2506.01015/) (2025)
- [Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer](https://mlsanity.com/paper/2605.30940/) (2026)
- [SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation](https://mlsanity.com/paper/2601.15017/) (2026)

## Abstract (arXiv)

Binaural audio generation (BAG) aims to convert monaural audio to stereo audio using visual prompts, requiring a deep understanding of spatial and semantic information. However, current models risk overfitting to room environments and lose fine-grained spatial details. In this paper, we propose a new audio-visual binaural generation model incorporating an audio-visual conditional normalisation layer that dynamically aligns the mean and variance of the target difference audio features using visual context, along with a new contrastive learning method to enhance spatial sensitivity by mining negative samples from shuffled visual features. We also introduce a cost-efficient way to utilise test-time augmentation in video data to enhance performance. Our approach achieves state-of-the-art generation accuracy on the FAIR-Play and MUSIC-Stereo benchmarks.