mlsanityData to 7 Oct 2026

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

arXiv 2102.03334 ↗First posted 5 Feb 2021stat.ML cs.LGcited by 1,224

Not on the map. Why?

Cited by 296 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

Key facts. Cited by 1,224 papers in the map (data through 7 Oct 2026). Builds on 50 of its 66 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

50 of 66 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2017Mask R-CNNcited by 7,259
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  16. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  19. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  20. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  21. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  22. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  23. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  24. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  25. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  26. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  27. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  28. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  29. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  30. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  31. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  32. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  33. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  34. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  35. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  36. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  37. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  38. 2020MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyondcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 244; 2023: 384; 2024: 284; 2025: 180; 2026: 75.

1,224 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
38 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  2. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  10. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  11. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  12. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  13. 2021OCR-free Document Understanding Transformercited by 400
  14. 2022Multimodal Learning with Transformers: A Surveycited by 399
  15. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  16. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  17. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  20. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  21. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  22. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  23. 2022Visual Spatial Reasoningcited by 288
  24. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  25. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  26. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  27. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  28. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  29. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  30. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  31. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  32. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  33. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  34. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  35. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  36. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  37. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  38. 2023Scalable 3D Captioning with Pretrained Modelscited by 201

50 most cited of 1,224 All 1,224 (JSON)