mlsanityData to 7 Oct 2026

UniT: Multimodal Multitask Learning with a Unified Transformer

arXiv 2102.10772 ↗First posted 22 Feb 2021cs.CV cs.CLcited by 195

Not on the map. Why?

Abstract (arXiv)

We propose UniT, a Unified Transformer model to simultaneously learn the most prominent tasks across different domains, ranging from object detection to natural language understanding and multimodal reasoning. Based on the transformer encoder-decoder architecture, our UniT model encodes each input modality with an encoder and makes predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads. The entire model is jointly trained end-to-end with losses from each task. Compared to previous efforts on multi-task learning with transformers, we share the same model parameters across all tasks instead of separately fine-tuning task-specific models and handle a much higher variety of tasks across different domains. In our experiments, we learn 7 tasks jointly over 8 datasets, achieving strong performance on each task with significantly fewer parameters. Our code is available in MMF at https://mmf.sh.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). Builds on 60 of its 73 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

60 of 73 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
38 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2017Non-local Neural Networkscited by 2,290
  13. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  14. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  15. 2020Point Transformercited by 1,462
  16. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  19. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  20. 2018Image Transformercited by 770
  21. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  22. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  23. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  24. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  25. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  26. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  27. 2019Attention Augmented Convolutional Networkscited by 363
  28. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  29. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  30. 2020Exploring Self-attention for Image Recognitioncited by 313
  31. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  32. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  33. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  34. 2019Local Relation Networks for Image Recognitioncited by 210
  35. 2018Gather-Excite: Exploiting Feature Context in Convolutional Neural Networkscited by 138
  36. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  37. 2020Robust Learning Through Cross-Task Consistencycited by 105
  38. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 56; 2023: 53; 2024: 34; 2025: 26; 2026: 7.

195 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  7. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  8. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  9. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  10. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  11. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  12. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
38 more
  1. 2022SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understandingcited by 164
  2. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  3. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  4. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  5. 20234M: Massively Multimodal Masked Modelingcited by 118
  6. 2021A Survey of Visual Transformerscited by 117
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  8. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  9. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  10. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  11. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  12. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  13. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  14. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  15. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  16. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  17. 2022IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenescited by 62
  18. 2023Towards Unified Token Learning for Vision-Language Trackingcited by 62
  19. 2023UniHCP: A Unified Model for Human-Centric Perceptionscited by 59
  20. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  21. 2022Where in the World is this Image? Transformer-based Geo-localization in the Wildcited by 58
  22. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  23. 2021Human-Adversarial Visual Question Answeringcited by 57
  24. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  25. 2023When Brain-inspired AI Meets AGIcited by 54
  26. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  27. 2022MulT: An End-to-End Multitask Learning Transformercited by 47
  28. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  29. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43
  30. 2025Omni-Video: Democratizing Unified Video Understanding and Generationcited by 39
  31. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 37
  32. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  33. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  34. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  35. 2022Multi-Task Learning with Multi-Query Transformer for Dense Predictioncited by 34
  36. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  37. 2023UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localizationcited by 33
  38. 2022Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-trainingcited by 32

50 most cited of 195 All 195 (JSON)