mlsanityData to 7 Oct 2026

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

arXiv 2006.06195 ↗First posted 11 Jun 2020cs.CV cs.CL cs.LGcited by 327

Not on the map. Why?

Cited by 89 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.

Key facts. Cited by 327 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 74; 2022: 87; 2023: 99; 2024: 43; 2025: 18; 2026: 6.

327 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  8. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
38 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  2. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  4. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  6. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2022Multimodal Learning with Transformers: A Surveycited by 399
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  13. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  14. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  15. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  16. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  17. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  18. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  19. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  20. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  21. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  22. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  23. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  24. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  25. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  26. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  27. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  28. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  29. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  30. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  31. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  32. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  33. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  34. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  35. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  36. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  37. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  38. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127

50 most cited of 327 All 327 (JSON)