mlsanityData to 7 Oct 2026

CCStereo: Audio-Visual Contextual and Contrastive Learning for Binaural Audio Generation

arXiv 2501.02786 ↗First posted 6 Jan 2025cs.SD cs.CV eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Binaural audio generation (BAG) aims to convert monaural audio to stereo audio using visual prompts, requiring a deep understanding of spatial and semantic information. However, current models risk overfitting to room environments and lose fine-grained spatial details. In this paper, we propose a new audio-visual binaural generation model incorporating an audio-visual conditional normalisation layer that dynamically aligns the mean and variance of the target difference audio features using visual context, along with a new contrastive learning method to enhance spatial sensitivity by mining negative samples from shuffled visual features. We also introduce a cost-efficient way to utilise test-time augmentation in video data to enhance performance. Our approach achieves state-of-the-art generation accuracy on the FAIR-Play and MUSIC-Stereo benchmarks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

34 of 51 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  9. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  12. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
22 more
  1. 2021Localizing Visual Sounds the Hard Waycited by 169
  2. 2013A Survey on Multi-view Learningcited by 152
  3. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  4. 2020Better Aggregation in Test-Time Augmentationcited by 121
  5. 2016A Survey of Multi-View Representation Learningcited by 107
  6. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  7. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  8. 2022Localizing Visual Sounds the Easy Waycited by 81
  9. 2018Batch-Instance Normalization for Adaptively Style-Invariant Neural Networkscited by 77
  10. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  11. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  12. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  13. 2020Pooling Methods in Deep Neural Networks, a Reviewcited by 52
  14. 2021Diverse Semantic Image Synthesis via Probability Distribution Modelingcited by 49
  15. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  16. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
  17. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 33
  18. 2024Understanding Test-Time Augmentationcited by 23
  19. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  20. 2021Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attentioncited by 17
  21. 2021Binaural Audio Generation via Multi-task Learningcited by 5
  22. 2023Cross-modal Generative Model for Visual-Guided Binaural Stereo Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  2. 2025AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Promptingcited by 2
  3. 2026SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generationcited by 0
  4. 2026Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformercited by 0