mlsanityData to 7 Oct 2026

Vision Transformers are Parameter-Efficient Audio-Visual Learners

arXiv 2212.07983 ↗First posted 15 Dec 2022cs.CV cs.CL cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Vision transformers (ViTs) have achieved impressive results on various computer vision tasks in the last several years. In this work, we study the capability of frozen ViTs, pretrained only on visual data, to generalize to audio-visual data without finetuning any of its original parameters. To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT. To efficiently fuse visual and audio cues, our LAVISH adapter uses a small set of latent tokens, which form an attention bottleneck, thus, eliminating the quadratic cost of standard cross-attention. Compared to the existing modality-specific audio-visual methods, our approach achieves competitive or even better performance on various audio-visual tasks while using fewer tunable parameters and without relying on costly audio pretraining or external audio encoders. Our code is available at https://genjib.github.io/project_page/LAVISH/

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 81 of its 100 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

81 of 100 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  12. 2021AST: Audio Spectrogram Transformercited by 634
  13. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  14. 2017Learning multiple visual domains with residual adapterscited by 553
  15. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  16. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  17. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  18. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  19. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  20. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  21. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  22. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  23. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  24. 2022Masked Autoencoders that Listencited by 271
  25. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  26. 2017Objects that Soundcited by 254
  27. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  28. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  29. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  30. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  31. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  32. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  33. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  34. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  35. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  36. 2021Localizing Visual Sounds the Hard Waycited by 169
  37. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  38. 2021Pretrained Transformers as Universal Computation Enginescited by 162

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 30; 2025: 19; 2026: 8.

72 papers.

Sort:

By area:

  1. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  2. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  3. 2023Annotation-free Audio-Visual Segmentationcited by 44
  4. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 39
  5. 2023Connecting Multi-modal Contrastive Representationscited by 36
  6. 2023Visual Tuningcited by 34
  7. 2023Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Taskscited by 31
  8. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  9. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  10. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  11. 2024IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFTcited by 26
  12. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
38 more
  1. 2023Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Modelcited by 25
  2. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  3. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  4. 2025Multimodal Referring Segmentation: A Surveycited by 22
  5. 2023MuseChat: A Conversational Music Recommendation System for Videoscited by 21
  6. 2023Extending Multi-modal Contrastive Representationscited by 21
  7. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  8. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  9. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  10. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  11. 2025Learning Musical Representations for Music Performance Question Answeringcited by 17
  12. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  13. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  14. 2024Boosting Audio Visual Question Answering via Key Semantic-Aware Cuescited by 14
  15. 2024Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependenciescited by 14
  16. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 14
  17. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  18. 2024Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approachcited by 12
  19. 2024Patch-level Sounding Object Tracking for Audio-Visual Question Answeringcited by 12
  20. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answeringcited by 12
  21. 2024MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformerscited by 11
  22. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  23. 2023Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarioscited by 8
  24. 2024Learning Modality Knowledge Alignment for Cross-Modality Transfercited by 8
  25. 2024Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendationcited by 7
  26. 2024Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognitioncited by 6
  27. 2024Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Modelcited by 6
  28. 2024SSFam: Scribble Supervised Salient Object Detection Familycited by 6
  29. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 6
  30. 2024Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognitioncited by 5
  31. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  32. 2023Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separationcited by 4
  33. 2024Answering Diverse Questions via Text Attached with Key Audio-Visual Cluescited by 4
  34. 2024CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answeringcited by 4
  35. 2024SAVE: Segment Audio-Visual Easy way using Segment Anything Modelcited by 4
  36. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  37. 2025PreFM: Online Audio-Visual Event Parsing via Predictive Future Modelingcited by 4
  38. 2025Learning Sparsity for Effective and Efficient Music Performance Question Answeringcited by 4

50 most cited of 72 All 72 (JSON)