mlsanityData to 14 Oct 2026

12-in-1: Multi-Task Vision and Language Representation Learning

arXiv 1912.02315 ↗First posted 5 Dec 2019cs.CV cs.CL cs.LGcited by 278

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly. In this work, we investigate these relationships between vision-and-language tasks by developing a large-scale, multi-task training regime. Our approach culminates in a single model on 12 datasets from four broad categories of task including visual question answering, caption-based image retrieval, grounding referring expressions, and multi-modal verification. Compared to independently trained single-task models, this represents a reduction from approximately 3 billion parameters to 270 million while simultaneously improving performance by 2.05 points on average across tasks. We use our multi-task framework to perform in-depth analysis of the effect of joint training diverse tasks. Further, we show that finetuning task-specific models from our single multi-task model can lead to further improvements, achieving performance at or above the state-of-the-art.

Key facts. Cited by 278 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 96; 2022: 95; 2023: 59; 2024: 16; 2025: 11; 2026: 1.

278 papers.

Sort:

By area:

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  12. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
38 more
  1. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  2. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  3. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  4. 2022Multimodal Learning with Transformers: A Surveycited by 400
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  7. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  8. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  9. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  10. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  11. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  12. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  13. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  14. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  15. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  16. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  17. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  18. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  19. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  20. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  21. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  22. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  23. 2022A Unified Sequence Interface for Vision Taskscited by 152
  24. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  25. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  26. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  27. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  28. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  29. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  30. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  31. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  32. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  33. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  34. 2021Causal Attention for Vision-Language Taskscited by 111
  35. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  36. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  37. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
  38. 2023Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Taskscited by 96

50 most cited of 278 All 278 (JSON)