mlsanityData to 14 Oct 2026

GIT: A Generative Image-to-text Transformer for Vision and Language

arXiv 2205.14100 ↗First posted 27 May 2022cs.CVcited by 537

Not on the map. Why?

Cited by 147 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at urlhttps://github.com/microsoft/GenerativeImage2Text.

Key facts. Cited by 537 papers in the map (data through 14 Oct 2026). Builds on 109 of its 149 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

109 of 149 references.

References · 109

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  14. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  17. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  18. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  19. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  21. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  22. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  23. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  24. 2016Self-critical Sequence Training for Image Captioningcited by 547
  25. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  28. 2018nocaps: novel object captioning at scalecited by 444
  29. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  30. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  31. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  32. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  33. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  34. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  35. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  36. 2019Scene Text Visual Question Answeringcited by 331
  37. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  38. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321

50 most cited of 109 All 109 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 29; 2023: 206; 2024: 150; 2025: 114; 2026: 38.

537 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  11. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
38 more
  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  7. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  9. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  10. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  12. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  13. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  14. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  15. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  16. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  17. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  18. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  19. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  20. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  21. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  22. 2023Video Understanding with Large Language Models: A Surveycited by 206
  23. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  24. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  25. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  26. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  27. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
  28. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  29. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 171
  30. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  31. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  32. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  33. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  34. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  35. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  36. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  37. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  38. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140

50 most cited of 537 All 537 (JSON)