mlsanityData to 7 Oct 2026

Unifying Vision-and-Language Tasks via Text Generation

arXiv 2102.02779 ↗First posted 4 Feb 2021cs.CL cs.AI cs.CVcited by 411

Not on the map. Why?

Cited by 105 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

Key facts. Cited by 411 papers in the map (data through 7 Oct 2026). Builds on 64 of its 86 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

64 of 86 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2017Mask R-CNNcited by 7,259
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  13. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2017Mixed Precision Trainingcited by 1,135
  16. 2018Self-Attention with Relative Position Representationscited by 1,112
  17. 2016Modeling Context in Referring Expressionscited by 1,054
  18. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  19. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  20. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  21. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  22. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  23. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  24. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  25. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  26. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  27. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  28. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  29. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  30. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  31. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  32. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  33. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  34. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  35. 2019Attention on Attention for Image Captioningcited by 289
  36. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  37. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  38. 2018Bilinear Attention Networkscited by 277

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 116; 2023: 137; 2024: 75; 2025: 46; 2026: 8.

411 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
38 more
  1. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  7. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  8. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  9. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  10. 2022Multimodal Learning with Transformers: A Surveycited by 399
  11. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  12. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  13. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  14. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  15. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  16. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  17. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  18. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  19. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  20. 2024Generative Representational Instruction Tuningcited by 227
  21. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  22. 2022Unified Structure Generation for Universal Information Extractioncited by 220
  23. 2022Learning Video Representations from Large Language Modelscited by 204
  24. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  25. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  26. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  27. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  28. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  29. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  30. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  31. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  32. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  33. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  34. 2022SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understandingcited by 164
  35. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  36. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  37. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  38. 2023Emu: Generative Pretraining in Multimodalitycited by 156

50 most cited of 411 All 411 (JSON)