mlsanityData to 7 Oct 2026

Deep Audio-Visual Learning: A Survey

arXiv 2001.04758 ↗First posted 14 Jan 2020cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 12; 2023: 20; 2024: 15; 2025: 7; 2026: 4.

68 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  3. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  4. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  5. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  6. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  7. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  8. 2021Responsive Listening Head Generation: A Benchmark Dataset and Baselinecited by 67
  9. 2021A Survey on Multimodal Disinformation Detectioncited by 47
  10. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  11. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 42
  12. 2024Diagnosing and Re-learning for Balanced Multimodal Learningcited by 40
38 more
  1. 2023Connecting Multi-modal Contrastive Representationscited by 36
  2. 2021Sound-Guided Semantic Image Manipulationcited by 35
  3. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  4. 2023Lightweight Vision Transformer with Bidirectional Interactioncited by 25
  5. 2023Physics-Driven Diffusion Models for Impact Sound Synthesis from Videoscited by 21
  6. 2024Faces that Speak: Jointly Synthesising Talking Face and Speech from Textcited by 21
  7. 2022Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPscited by 20
  8. 2022UAVM: Towards Unifying Audio and Visual Modelscited by 20
  9. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  10. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 15
  11. 2023Interactive Conversational Head Generationcited by 14
  12. 2023Neural Semantic Surface Mapscited by 13
  13. 2023Audio-Visual Speech Enhancement with Score-Based Generative Modelscited by 12
  14. 2024StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Headscited by 11
  15. 2022CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognitioncited by 10
  16. 2024Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localizationcited by 10
  17. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  18. 2022A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!cited by 9
  19. 2023UniFLG: Unified Facial Landmark Generator from Text or Speechcited by 8
  20. 2023That's What I Said: Fully-Controllable Talking Face Generationcited by 8
  21. 2023LAVSS: Location-Guided Audio-Visual Spatial Audio Separationcited by 8
  22. 2022Noise-Tolerant Learning for Audio-Visual Action Recognitioncited by 7
  23. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  24. 2021TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separationcited by 6
  25. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 6
  26. 2021AudioVisual Video Summarizationcited by 5
  27. 2021OpenFed: A Comprehensive and Versatile Open-Source Federated Learning Frameworkcited by 5
  28. 2023Versatile audio-visual learning for emotion recognitioncited by 5
  29. 2026LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoningcited by 5
  30. 2023Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Modelcited by 4
  31. 2023Hierarchical Semantic Perceptual Listener Head Video Generation: A High-performance Pipelinecited by 4
  32. 2024Attend-Fusion: Efficient Audio-Visual Fusion for Video Classificationcited by 3
  33. 2024Unveiling Visual Biases in Audio-Visual Localization Benchmarkscited by 3
  34. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  35. 2025From Waveforms to Pixels: A Survey on Audio-Visual Segmentationcited by 3
  36. 2021Music source separation conditioned on 3D point cloudscited by 2
  37. 2022SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidancecited by 2
  38. 2023Visually-Guided Sound Source Separation with Audio-Visual Predictive Codingcited by 2

50 most cited of 68 All 68 (JSON)