mlsanityData to 14 Oct 2026

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

arXiv 2208.10442 ↗First posted 22 Aug 2022cs.CV cs.CLcited by 511

Not on the map. Why?

Cited by 130 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).

Key facts. Cited by 511 papers in the map (data through 14 Oct 2026). Builds on 59 of its 75 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

59 of 75 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  15. 2021Grounded Language-Image Pre-trainingcited by 1,156
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  18. 2021Scaling Vision Transformerscited by 999
  19. 2016Deep Networks with Stochastic Depthcited by 979
  20. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  21. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  22. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  23. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  24. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  25. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  26. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  27. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  28. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  29. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  30. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  31. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  32. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 429
  33. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  34. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 382
  35. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  36. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  37. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 316
  38. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 70; 2023: 239; 2024: 132; 2025: 62; 2026: 8.

511 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  9. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  10. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  11. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
38 more
  1. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  2. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  5. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  6. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  7. 2022MAGVIT: Masked Generative Video Transformercited by 346
  8. 2023ClimaX: A foundation model for weather and climatecited by 337
  9. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  10. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  11. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  12. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  13. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  14. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  15. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 274
  16. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  17. 2023A Cookbook of Self-Supervised Learningcited by 250
  18. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  19. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  20. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  21. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  22. 2022Learning Video Representations from Large Language Modelscited by 205
  23. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  24. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  25. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  26. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  27. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  28. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  29. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  30. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  31. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  32. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  33. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 151
  34. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  35. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  36. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  37. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  38. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 134

50 most cited of 511 All 511 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
  2. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  4. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
  5. 2023Vision Language Transformers: A Survey
  6. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  7. 2022A Survey of Vision-Language Pre-Trained Models
  8. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretraining