mlsanityData to 14 Oct 2026

UAVM: Towards Unifying Audio and Visual Models

arXiv 2208.00061 ↗First posted 29 Jul 2022cs.CV cs.MM cs.SDcited by 20

Not on the map. Why?

Abstract (arXiv)

Conventional audio-visual models have independent audio and video branches. In this work, we unify the audio and visual branches by designing a Unified Audio-Visual Model (UAVM). The UAVM achieves a new state-of-the-art audio-visual event classification accuracy of 65.8% on VGGSound. More interestingly, we also find a few intriguing properties of UAVM that the modality-independent counterparts do not have.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 29 Jul 2022.

Builds on

28 of 31 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2022A ConvNet for the 2020scited by 3,579
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2021AST: Audio Spectrogram Transformercited by 643
  12. 2017Look, Listen and Learncited by 458
16 more
  1. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  5. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  7. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  8. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
  9. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 82
  10. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  11. 2020Deep Audio-Visual Learning: A Surveycited by 69
  12. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  13. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  14. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  15. 2020Large Scale Audiovisual Learning of Sounds with Weakly Labeled Datacited by 27
  16. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 8; 2026: 1.

20 papers.

Sort:

By area:

  1. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  2. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  3. 2024Audio Mamba: Bidirectional State Space Model for Audio Representation Learningcited by 37
  4. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  5. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 14
  6. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  7. 2023Versatile audio-visual learning for emotion recognitioncited by 5
  8. 2024Triple Disentangled Representation Learning for Multimodal Affective Analysiscited by 4
  9. 2023Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learningcited by 3
  10. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  11. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  12. 2023AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codescited by 1
8 more
  1. 2024Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognitioncited by 1
  2. 2025SNIFR : Boosting Fine-Grained Child Harmful Content Detection Through Audio-Visual Alignment with Cascaded Cross-Transformercited by 1
  3. 2026Audio-Visual Continual Test-Time Adaptation without Forgettingcited by 1
  4. 2024Multimodal Fusion via Self-Consistent Task-Gradient Fieldscited by 0
  5. 2025SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verificationcited by 0
  6. 2025Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognitioncited by 0
  7. 2025Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voicecited by 0
  8. 2025MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimizationcited by 0