mlsanityData to 14 Oct 2026

Accommodating Audio Modality in CLIP for Multimodal Processing

arXiv 2303.06591 ↗First posted 12 Mar 2023cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multimodal processing has attracted much attention lately especially with the success of pre-training. However, the exploration has mainly focused on vision-language pre-training, as introducing more modalities can greatly complicate model design and optimization. In this paper, we extend the stateof-the-art Vision-Language model CLIP to accommodate the audio modality for Vision-Language-Audio multimodal processing. Specifically, we apply inter-modal and intra-modal contrastive learning to explore the correlation between audio and other modalities in addition to the inner characteristics of the audio modality. Moreover, we further design an audio type token to dynamically learn different audio information type for different scenarios, as both verbal and nonverbal heterogeneous information is conveyed in general audios. Our proposed CLIP4VLA model is validated in different downstream tasks including video retrieval and video captioning, and achieves the state-of-the-art performance on the benchmark datasets of MSR-VTT, VATEX, and Audiocaps.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 46 references that are in the map. First posted on arXiv 12 Mar 2023.

Builds on

36 of 46 references.

References · 36

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  7. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  8. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  9. 2021AST: Audio Spectrogram Transformercited by 643
  10. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  12. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
24 more
  1. 2020Multi-modal Transformer for Video Retrievalcited by 395
  2. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  3. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  4. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  5. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  6. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  7. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  8. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  9. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  12. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  13. 2016Exploring the Limits of Language Modelingcited by 173
  14. 2020Support-set bottlenecks for video-text representation learningcited by 170
  15. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 107
  16. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  17. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  18. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  19. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  20. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  21. 2020ESResNet: Environmental Sound Classification Based on Visual Domain Modelscited by 31
  22. 2021OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generationcited by 20
  23. 2021Cross-Modal Graph with Meta Concepts for Video Captioningcited by 4
  24. 2021What Matters for Ad-hoc Video Search? A Large-scale Evaluation on TRECVIDcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 8; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024Gramian Multimodal Representation Learning and Alignmentcited by 54
  2. 2024Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalitiescited by 24
  3. 2025Principled Multimodal Representation Learningcited by 19
  4. 2023Audio Generation with Multiple Conditional Diffusion Modelcited by 18
  5. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 18
  6. 2026DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representationcited by 5
  7. 2026Towards Uniformity and Alignment for Multimodal Representation Learningcited by 4
  8. 2023Exploring the Role of Audio in Video Captioningcited by 3
  9. 2025TRIDENT: Tri-Modal Molecular Representation Learning with Taxonomic Annotations and Local Correspondencecited by 3
  10. 2025Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learningcited by 1
  11. 2025Leveraging CLIP Encoder for Multimodal Emotion Recognitioncited by 1
  12. 2025Contrastive Pretraining with Dual Visual Encoders for Gloss-Free Sign Language Translationcited by 1
6 more
  1. 2024STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraftcited by 0
  2. 2025FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoderscited by 0
  3. 2025Multimodal Functional Maximum Correlation for Emotion Recognitioncited by 0
  4. 2026What Images Cannot Say: Language-Guided Olfactory Representation Learningcited by 0
  5. 2026Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessmentcited by 0
  6. 2026Hypergraph-Regularized Gramian Volumes for Multimodal Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
  2. 2025Closing the Modality Gap for Mixed Modality Search
  3. 2026Empowering Long-form Omni-modal Understanding with Robust Audio Perception
  4. 2022A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval
  5. 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
  6. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  7. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
  8. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion