mlsanityData to 14 Oct 2026

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

arXiv 2305.18500 ↗First posted 29 May 2023cs.CV cs.AI cs.CLcited by 169

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). Builds on 88 of its 102 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

88 of 102 references.

References · 88

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2017The Kinetics Human Action Video Datasetcited by 1,909
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  12. 2017Dense-Captioning Events in Videoscited by 793
  13. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  15. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  16. 2017Localizing Moments in Video with Natural Languagecited by 663
  17. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  19. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  20. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  21. 2016Self-critical Sequence Training for Image Captioningcited by 547
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  23. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  24. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  25. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  26. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  27. 2020Multi-modal Transformer for Video Retrievalcited by 395
  28. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  29. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  30. 2019Clotho: An Audio Captioning Datasetcited by 358
  31. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  32. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  33. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  34. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  35. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  36. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  37. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  38. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 44; 2025: 68; 2026: 41.

169 papers.

Sort:

By area:

  1. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  2. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  3. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  4. 2023Video Understanding with Large Language Models: A Surveycited by 206
  5. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  6. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  7. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  8. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  9. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  10. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  11. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  12. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
38 more
  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  2. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  3. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  4. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  5. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  6. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  7. 2024Gramian Multimodal Representation Learning and Alignmentcited by 54
  8. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  9. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 51
  10. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  11. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  12. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  13. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 36
  14. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  15. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 35
  16. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 35
  17. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  18. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  19. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  20. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 27
  21. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  22. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  23. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 25
  24. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24
  25. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  26. 2024EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrievalcited by 21
  27. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  28. 2025Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrievalcited by 20
  29. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  30. 2025Principled Multimodal Representation Learningcited by 19
  31. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 18
  32. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  33. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  34. 2024MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrievalcited by 17
  35. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reductioncited by 15
  36. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  37. 2024Do Language Models Understand Time?cited by 14
  38. 2025HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronizationcited by 14

50 most cited of 169 All 169 (JSON)