mlsanityData to 7 Oct 2026

Recent Advances and Challenges in Deep Audio-Visual Correlation Learning

arXiv 2202.13673 ↗First posted 28 Feb 2022cs.MM cs.CV cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Audio-visual correlation learning aims to capture essential correspondences and understand natural phenomena between audio and video. With the rapid growth of deep learning, an increasing amount of attention has been paid to this emerging research issue. Through the past few years, various methods and datasets have been proposed for audio-visual correlation learning, which motivate us to conclude a comprehensive survey. This survey paper focuses on state-of-the-art (SOTA) models used to learn correlations between audio and video, but also discusses some tasks of definition and paradigm applied in AI multimedia. In addition, we investigate some objective functions frequently used for optimizing audio-visual correlation learning models and discuss how audio-visual data is exploited in the optimization process. Most importantly, we provide an extensive comparison and summarization of the recent progress of SOTA audio-visual correlation learning and discuss future research directions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 115 of its 247 references that are in the map. First posted on arXiv 28 Feb 2022.

Builds on

115 of 247 references.

References · 115

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017Neural Discrete Representation Learningcited by 4,650
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2021ViViT: A Video Vision Transformercited by 1,610
  8. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  11. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  12. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  13. 2021AST: Audio Spectrogram Transformercited by 634
  14. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  15. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  16. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  17. 2020Jukebox: A Generative Model for Musiccited by 535
  18. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  19. 2017Look, Listen and Learncited by 452
  20. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  21. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  22. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  23. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  24. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  25. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  26. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  27. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  28. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  29. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  30. 2018The Sound of Pixelscited by 258
  31. 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
  32. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  33. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  34. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  35. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  36. 2021Localizing Visual Sounds the Hard Waycited by 169
  37. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  38. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168

50 most cited of 115 All 115 (JSON)

Built on by

Not cited yet.