mlsanityData to 14 Oct 2026

VGGSounder: Audio-Visual Evaluations for Foundation Models

arXiv 2508.08237 ↗First posted 11 Aug 2025cs.MM cs.AI cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding. The VGGSound dataset is commonly used as a benchmark for evaluation audio-visual classification. However, our analysis identifies several limitations of VGGSound, including incomplete labelling, partially overlapping classes, and misaligned modalities. These lead to distorted evaluations of auditory and visual capabilities. To address these limitations, we introduce VGGSounder, a comprehensively re-annotated, multi-label test set that extends VGGSound and is specifically designed to evaluate audio-visual foundation models. VGGSounder features detailed modality annotations, enabling precise analyses of modality-specific performance. Furthermore, we reveal model limitations by analysing performance degradation when adding another input modality with our new modality confusion metric.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 76 of its 94 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

76 of 94 references.

References · 76

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2017The Kinetics Human Action Video Datasetcited by 1,909
  5. 2024A Survey on LLM-as-a-Judgecited by 1,423
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  9. 2020Aligning AI With Shared Human Valuescited by 723
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2017Look, Listen and Learncited by 458
  12. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
38 more
  1. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  4. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  5. 2020Are we done with ImageNet?cited by 333
  6. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  7. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  8. 2018Deep Audio-Visual Speech Recognitioncited by 322
  9. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  11. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  12. 2018The Sound of Pixelscited by 259
  13. 2017Objects that Soundcited by 256
  14. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  15. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  16. 2024Are We Done with MMLU?cited by 194
  17. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  18. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  19. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  20. 2021Localizing Visual Sounds the Hard Waycited by 170
  21. 2019The Sound of Motionscited by 146
  22. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 146
  23. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  24. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  25. 2019Co-Separating Sounds of Visual Objectscited by 110
  26. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  27. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  28. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 97
  29. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  30. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  31. 2020Foley Music: Learning to Generate Music from Videoscited by 83
  32. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  33. 2022Localizing Visual Sounds the Easy Waycited by 82
  34. 2016Ambient Sound Provides Supervision for Visual Learningcited by 78
  35. 2019Dual-modality seq2seq network for audio-visual event localizationcited by 72
  36. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  37. 2020Generating Visually Aligned Sound from Videoscited by 67
  38. 2020Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learningcited by 63

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 20
  2. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generationcited by 11
  3. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 5
  4. 2025Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generationcited by 1
  5. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  6. 2026Auditing Frame-Level AUC in Weakly Supervised Video Anomaly Detection: Granularity, Resolution, and Scene Biascited by 1
  7. 2026Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalizationcited by 0
  8. 2026Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Frameworkcited by 0