A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning
Not on the map. Why?
Abstract (arXiv)
Audio-visual correlation learning aims to capture and understand natural phenomena between audio and visual data. The rapid growth of Deep Learning propelled the development of proposals that process audio-visual data and can be observed in the number of proposals in the past years. Thus encouraging the development of a comprehensive survey. Besides analyzing the models used in this context, we also discuss some tasks of definition and paradigm applied in AI multimedia. In addition, we investigate objective functions frequently used and discuss how audio-visual data is exploited in the optimization process, i.e., the different methodologies for representing knowledge in the audio-visual domain. In fact, we focus on how human-understandable mechanisms, i.e., structured knowledge that reflects comprehensible knowledge, can guide the learning process. Most importantly, we provide a summarization of the recent progress of Audio-Visual Correlation Learning (AVCL) and discuss the future research directions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 145 of its 247 references that are in the map. First posted on arXiv 24 Nov 2024.
Builds on
145 of 247 references.
References · 145
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2017Neural Discrete Representation Learningcited by 4,650
38 more
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2021ViViT: A Video Vision Transformercited by 1,610
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
- 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
- 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
- 2021AST: Audio Spectrogram Transformercited by 634
- 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
- 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
- 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
- 2020Jukebox: A Generative Model for Musiccited by 535
- 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
- 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
- 2017Look, Listen and Learncited by 452
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
- 2022Multimodal Learning with Transformers: A Surveycited by 399
- 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
- 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
- 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
- 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
- 2021Attention Bottlenecks for Multimodal Fusioncited by 361
- 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
- 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
- 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
- 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
- 2020Self-Supervised MultiModal Versatile Networkscited by 276
- 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
- 2018The Sound of Pixelscited by 258
- 2023Multimodal Deep Learningcited by 257
- 2015Beyond Short Snippets: Deep Networks for Video Classificationcited by 233
50 most cited of 145 All 145 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.