mlsanityData to 7 Oct 2026

UFO: A UniFied TransfOrmer for Vision-Language Representation Learning

arXiv 2111.10023 ↗First posted 19 Nov 2021cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a single UniFied transfOrmer (UFO), which is capable of processing either unimodal inputs (e.g., image or language) or multimodal inputs (e.g., the concatenation of the image and the question), for vision-language (VL) representation learning. Existing approaches typically design an individual network for each modality and/or a specific fusion network for multimodal tasks. To simplify the network architecture, we use a single transformer network and enforce multi-task learning during VL pre-training, which includes the image-text contrastive loss, image-text matching loss, and masked language modeling loss based on the bidirectional and the seq2seq attention mask. The same transformer network is used as the image encoder, the text encoder, or the fusion network in different pre-training tasks. Empirically, we observe less conflict among different tasks and achieve new state of the arts on visual question answering, COCO image captioning (cross-entropy optimization) and nocaps (in SPICE). On other downstream tasks, e.g., image-text retrieval, we also achieve competitive performance.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 46 of its 55 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

46 of 55 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
34 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  13. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  14. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  15. 2016Self-critical Sequence Training for Image Captioningcited by 544
  16. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  18. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  21. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  22. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  23. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  24. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  25. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  26. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  27. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  28. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  29. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  30. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  31. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  32. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  33. 2021LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrievalcited by 44
  34. 2020VIVO: Visual Vocabulary Pre-Training for Novel Object Captioningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 19; 2024: 3; 2025: 0; 2026: 1.

45 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  5. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  6. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  7. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  8. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  9. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  10. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
33 more
  1. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  2. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  3. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  4. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  5. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  6. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  7. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  8. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 37
  9. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  10. 2022FashionViL: Fashion-Focused Vision-and-Language Representation Learningcited by 32
  11. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32
  12. 2022CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrievalcited by 25
  13. 2022Unifying Vision-Language Representation Space with Single-tower Transformercited by 21
  14. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignmentcited by 19
  15. 2024TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioningcited by 17
  16. 2022Exploring Discrete Diffusion Models for Image Captioningcited by 16
  17. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  18. 2023Towards Models that Can See and Readcited by 15
  19. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  20. 2024Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptationcited by 13
  21. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  22. 2023"Let's not Quote out of Context": Unified Vision-Language Pretraining for Context Assisted Image Captioningcited by 7
  23. 2024PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adaptercited by 7
  24. 2022LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videoscited by 6
  25. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentationcited by 6
  26. 2022Learning to translate by learning to communicatecited by 4
  27. 2023The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluationcited by 4
  28. 2023Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Promptscited by 3
  29. 2023Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Taskscited by 2
  30. 2023Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regimecited by 1
  31. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimizationcited by 1
  32. 2023Unified Contrastive Fusion Transformer for Multimodal Human Action Recognitioncited by 0
  33. 2023E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizercited by 0