mlsanityData to 7 Oct 2026

Learning Audio-Video Modalities from Image Captions

arXiv 2204.00679 ↗First posted 1 Apr 2022cs.CV cs.MM cs.SDcited by 80

Not on the map. Why?

Abstract (arXiv)

A major challenge in text-video and text-audio retrieval is the lack of large-scale training data. This is unlike image-captioning, where datasets are in the order of millions of samples. To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort. Using this pipeline, we create a new large-scale, weakly labelled audio-video captioning dataset consisting of millions of paired clips and captions. We show that training a multimodal transformed based model on this data achieves competitive performance on video retrieval and video captioning, matching or even outperforming HowTo100M pretraining with 20x fewer clips. We also show that our mined clips are suitable for text-audio pretraining, and achieve state of the art results for the task of audio retrieval.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 67 of its 89 references that are in the map. First posted on arXiv 1 Apr 2022.

Builds on

67 of 89 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
38 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2017Localizing Moments in Video with Natural Languagecited by 660
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  5. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  6. 2021AST: Audio Spectrogram Transformercited by 634
  7. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  9. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  10. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  13. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  14. 2020Multi-modal Transformer for Video Retrievalcited by 395
  15. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  16. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  17. 2019Clotho: An Audio Captioning Datasetcited by 354
  18. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  19. 2016Image Captioning with Semantic Attentioncited by 260
  20. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  21. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  22. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  23. 2016Movie Descriptioncited by 180
  24. 2015Cross Modal Distillation for Supervision Transfercited by 173
  25. 2020Support-set bottlenecks for video-text representation learningcited by 170
  26. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  27. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  28. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  29. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  30. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  31. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  32. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  33. 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 106
  34. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  35. 2020Condensed Movies: Story Based Retrieval with Contextual Embeddingscited by 95
  36. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  37. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  38. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 31; 2024: 20; 2025: 17; 2026: 3.

80 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  3. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  4. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  5. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  6. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  8. 2022Learning Video Representations from Large Language Modelscited by 204
  9. 2023A Survey on Video Diffusion Modelscited by 203
  10. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  11. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  12. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
38 more
  1. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  2. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  3. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  4. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  5. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  7. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  8. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  9. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  10. 2023Audio-Visual LLM for Video Understandingcited by 76
  11. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  12. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  13. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  14. 2023AutoAD: Movie Description in Contextcited by 50
  15. 2023Data Roaming and Quality Assessment for Composed Image Retrievalcited by 45
  16. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  17. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  18. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  19. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  20. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  21. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  22. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  23. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  24. 2023Correlation-Guided Query-Dependency Calibration for Video Temporal Groundingcited by 31
  25. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  26. 2024Taming Data and Transformers for Audio Generationcited by 28
  27. 2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 28
  28. 2023Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Modelscited by 27
  29. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  30. 2024Saliency-Guided DETR for Moment Retrieval and Highlight Detectioncited by 21
  31. 2024Distilling Vision-Language Models on Millions of Videoscited by 20
  32. 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perceptioncited by 19
  33. 2025Principled Multimodal Representation Learningcited by 19
  34. 2022DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Datasetcited by 17
  35. 2024Sakuga-42M Dataset: Scaling Up Cartoon Researchcited by 17
  36. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 17
  37. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 15
  38. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13

50 most cited of 80 All 80 (JSON)