mlsanityData to 14 Oct 2026

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

arXiv 1908.02265 ↗First posted 6 Aug 2019cs.CV cs.CLcited by 2,222

Not on the map. Why?

Cited by 388 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

Key facts. Cited by 2,222 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 421; 2022: 522; 2023: 518; 2024: 360; 2025: 269; 2026: 131.

2,222 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2021Grounded Language-Image Pre-trainingcited by 1,156
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  8. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  9. 2021Transformers in Vision: A Surveycited by 1,013
  10. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  11. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  12. 2021Multiscale Vision Transformerscited by 897
  13. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  15. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  17. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  18. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  19. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  20. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  21. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  22. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  23. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  24. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  25. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  27. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  28. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  29. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  30. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  31. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  32. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  33. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  34. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  35. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  36. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  37. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  38. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431

50 most cited of 2,222 All 2,222 (JSON)