mlsanityData to 7 Oct 2026

FLAVA: A Foundational Language And Vision Alignment Model

arXiv 2112.04482 ↗First posted 8 Dec 2021cs.CV cs.CLcited by 646

Not on the map. Why?

Cited by 212 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a "foundation", that targets all modalities at once -- a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.

Key facts. Cited by 646 papers in the map (data through 7 Oct 2026). Builds on 90 of its 128 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

90 of 128 references.

References · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2016Layer Normalizationcited by 5,579
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  13. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  16. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  17. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  18. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  19. 2013Describing Textures in the Wildcited by 2,110
  20. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  21. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  22. 2019Towards VQA Models That Can Readcited by 1,696
  23. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  24. 2021ViViT: A Video Vision Transformercited by 1,610
  25. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  26. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  27. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  28. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  29. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  30. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  31. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  32. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  33. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  34. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  35. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  36. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  37. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  38. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 96; 2023: 193; 2024: 188; 2025: 116; 2026: 53.

646 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  10. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  11. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  12. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
38 more
  1. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Visual Classification via Description from Large Language Modelscited by 331
  4. 2024What matters when building vision-language models?cited by 321
  5. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  6. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  7. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  10. 2023A Cookbook of Self-Supervised Learningcited by 248
  11. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  12. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  13. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  14. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  15. 2024Generative Representational Instruction Tuningcited by 227
  16. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  17. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  18. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  19. 2023Teaching CLIP to Count to Tencited by 194
  20. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  21. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  22. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  23. 2023Language-Driven Representation Learning for Roboticscited by 163
  24. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 162
  25. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  26. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  27. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  28. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  29. 2023PACO: Parts and Attributes of Common Objectscited by 146
  30. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  31. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  32. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  33. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  34. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  35. 20234M: Massively Multimodal Masked Modelingcited by 118
  36. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  37. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  38. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115

50 most cited of 646 All 646 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Assessing and Learning Alignment of Unimodal Vision and Language Models
  2. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
  3. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  4. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  5. 2025Vision Language Models: A Survey of 26K Papers
  6. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  7. 2023Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions