# Collaborative Learning to Generate Audio-Video Jointly

arXiv: https://arxiv.org/abs/2104.02656 · HTML: https://mlsanity.com/paper/2104.02656/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Generating Visually Aligned Sound from Videos](https://mlsanity.com/paper/2008.00820/) (2020)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Speech2Face: Learning the Face Behind a Voice](https://mlsanity.com/paper/1905.09773/) (2019)
- [Audeo: Audio Generation for a Silent Performance Video](https://mlsanity.com/paper/2006.14348/) (2020)
- [CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation](https://mlsanity.com/paper/1711.08102/) (2017)
- [Sound2Sight: Generating Visual Dynamics from Sound and Context](https://mlsanity.com/paper/2007.12130/) (2020)
- [Towards Audio to Scene Image Synthesis using Generative Adversarial Network](https://mlsanity.com/paper/1808.04108/) (2018)
- [Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition](https://mlsanity.com/paper/1711.05747/) (2017)
- [Triangle Generative Adversarial Networks](https://mlsanity.com/paper/1709.06548/) (2017)
- [Conditional WaveGAN](https://mlsanity.com/paper/1809.10636/) (2018)

## Built on by (most cited first, 6 of 6)

- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Video-to-Audio Generation with Hidden Alignment](https://mlsanity.com/paper/2407.07464/) (2024)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [TA-V2A: Textually Assisted Video-to-Audio Generation](https://mlsanity.com/paper/2503.10700/) (2025)
- [Sensor-invariant Fingerprint ROI Segmentation Using Recurrent Adversarial Learning](https://mlsanity.com/paper/2107.01361/) (2021)

## Abstract (arXiv)

There have been a number of techniques that have demonstrated the generation of multimedia data for one modality at a time using GANs, such as the ability to generate images, videos, and audio. However, so far, the task of multi-modal generation of data, specifically for audio and videos both, has not been sufficiently well-explored. Towards this, we propose a method that demonstrates that we are able to generate naturalistic samples of video and audio data by the joint correlated generation of audio and video modalities. The proposed method uses multiple discriminators to ensure that the audio, video, and the joint output are also indistinguishable from real-world samples. We present a dataset for this task and show that we are able to generate realistic samples. This method is validated using various standard metrics such as Inception Score, Frechet Inception Distance (FID) and through human evaluation.