mlsanityData to 7 Oct 2026

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

arXiv 2304.08345 ↗First posted 17 Apr 2023cs.LG cs.CL cs.CVcited by 138

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 106 of its 134 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

106 of 134 references.

References · 106

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2021Video Swin Transformercited by 1,090
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  15. 2017Dense-Captioning Events in Videoscited by 789
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  18. 2017Localizing Moments in Video with Natural Languagecited by 660
  19. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  21. 2021AST: Audio Spectrogram Transformercited by 634
  22. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  23. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  24. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  25. 2016Self-critical Sequence Training for Image Captioningcited by 544
  26. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  27. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  28. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  29. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  30. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  31. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  32. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  33. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  34. 2020Multi-modal Transformer for Video Retrievalcited by 395
  35. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  36. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  37. 2019Clotho: An Audio Captioning Datasetcited by 354
  38. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 29; 2024: 38; 2025: 47; 2026: 24.

138 papers.

Sort:

By area:

  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  4. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  5. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  8. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  9. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  10. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  11. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  12. 2023Audio-Visual LLM for Video Understandingcited by 76
38 more
  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  3. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  4. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  5. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  6. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  7. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  8. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  9. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  10. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  11. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  12. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  13. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  14. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  15. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  16. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  17. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  18. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  19. 2023Parameter-efficient Tuning of Large-scale Multimodal Foundation Modelcited by 23
  20. 2024Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalitiescited by 23
  21. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  22. 2025Principled Multimodal Representation Learningcited by 19
  23. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
  24. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  25. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  26. 2024MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrievalcited by 17
  27. 2024Tri-Modal Motion Retrieval by Learning a Joint Embedding Spacecited by 16
  28. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  29. 2024On the Audio Hallucinations in Large Audio-Video Language Modelscited by 15
  30. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  31. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 15
  32. 2024Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answeringcited by 14
  33. 2024Boosting Audio Visual Question Answering via Key Semantic-Aware Cuescited by 14
  34. 2025HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronizationcited by 14
  35. 2024VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Modelcited by 12
  36. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 12
  37. 2024Patch-level Sounding Object Tracking for Audio-Visual Question Answeringcited by 12
  38. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answeringcited by 12

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
  2. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
  3. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
  4. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
  5. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  6. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
  7. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks