mlsanityData to 7 Oct 2026

MulT: An End-to-End Multitask Learning Transformer

arXiv 2205.08303 ↗First posted 17 May 2022cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

We propose an end-to-end Multitask Learning Transformer framework, named MulT, to simultaneously learn multiple high-level vision tasks, including depth estimation, semantic segmentation, reshading, surface normal estimation, 2D keypoint detection, and edge detection. Based on the Swin transformer model, our framework encodes the input image into a shared representation and makes predictions for each vision task using task-specific transformer-based decoder heads. At the heart of our approach is a shared attention mechanism modeling the dependencies across the tasks. We evaluate our model on several multitask benchmarks, showing that our MulT framework outperforms both the state-of-the art multitask convolutional neural network models and all the respective single task transformer models. Our experiments further highlight the benefits of sharing attention across all the tasks, and demonstrate that our MulT model is robust and generalizes well to new domains. Our project website is at https://ivrl.github.io/MulT/.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 52 of its 65 references that are in the map. First posted on arXiv 17 May 2022.

Builds on

52 of 65 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2017Non-local Neural Networkscited by 2,290
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020Pre-Trained Image Processing Transformercited by 891
  8. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  9. 2017A Survey on Multi-Task Learningcited by 838
  10. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  11. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  12. 2018Image Transformercited by 770
  13. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  14. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  15. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  16. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  17. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  18. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  19. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  20. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  21. 2019Attention Augmented Convolutional Networkscited by 363
  22. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  23. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  24. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  25. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  26. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  27. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  28. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  29. 2020Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformerscited by 190
  30. 2021Are Transformers More Robust Than CNNs?cited by 187
  31. 2018Gather-Excite: Exploiting Feature Context in Convolutional Neural Networkscited by 138
  32. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  33. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  34. 2020Robust Learning Through Cross-Task Consistencycited by 105
  35. 2017Measuring the tendency of CNNs to Learn Surface Statistical Regularitiescited by 87
  36. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  37. 2019Many Task Learning with Task Routingcited by 50
  38. 2019Zero-Shot Task Transfercited by 26

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 10; 2025: 12; 2026: 1.

47 papers.

Sort:

By area:

  1. 20234M: Massively Multimodal Masked Modelingcited by 118
  2. 2023DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Predictioncited by 27
  3. 2023InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understandingcited by 22
  4. 2023UniRef++: Segment Every Reference Object in Spatial and Temporal Spacescited by 22
  5. 2024DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Datacited by 18
  6. 2025Selective Task Group Updates for Multi-Task Optimizationcited by 15
  7. 2022PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Datacited by 14
  8. 2023Vision Transformer Adapters for Generalizable Multitask Learningcited by 14
  9. 2023PolyMaX: General Dense Prediction with Mask Transformercited by 14
  10. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  11. 2023Semantic Human Parsing via Scalable Semantic Transfer over Multiple Label Domainscited by 11
  12. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
35 more
  1. 2023Efficient Computation Sharing for Multi-Task Visual Scene Understandingcited by 8
  2. 2023Episodic Multi-Task Learning with Heterogeneous Neural Processescited by 8
  3. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
  4. 2024One missing piece in Vision and Language: A Survey on Comics Understandingcited by 8
  5. 2023Efficient Controllable Multi-Task Architecturescited by 7
  6. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  7. 2023AdaMTL: Adaptive Input-dependent Inference for Efficient Multi-Task Learningcited by 5
  8. 2023Multi-task View Synthesis with Neural Radiance Fieldscited by 5
  9. 2025TADFormer : Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learningcited by 5
  10. 2023MTLSegFormer: Multi-task Learning with Transformers for Semantic Segmentation in Precision Agriculturecited by 3
  11. 2024Elite360M: Efficient 360 Multi-task Learning via Bi-projection Fusion and Cross-task Collaborationcited by 3
  12. 2025UGoDIT: Unsupervised Group Deep Image Prior Via Transferable Weightscited by 3
  13. 2025RealDriveSim: A Realistic Multi-Modal Multi-Task Synthetic Dataset for Autonomous Drivingcited by 3
  14. 2025Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learningcited by 3
  15. 2023Multi-Similarity Contrastive Learningcited by 2
  16. 2023DeGMix: Efficient Multi-Task Dense Prediction with Deformable and Gating Mixercited by 2
  17. 2024SGW-based Multi-Task Learning in Vision Taskscited by 2
  18. 2023Dense Multitask Learning to Reconfigure Comicscited by 1
  19. 2024InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learningcited by 1
  20. 2024Multi-Cohort Framework with Cohort-Aware Attention and Adversarial Mutual-Information Minimization for Whole Slide Image Classificationcited by 1
  21. 2025Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Predictioncited by 1
  22. 2022Improving Long-tailed Object Detection with Image-Level Supervision by Multi-Task Collaborative Learningcited by 0
  23. 2023Joint 2D-3D Multi-Task Learning on Cityscapes-3D: 3D Detection, Segmentation, and Depth Estimationcited by 0
  24. 2023AutoTaskFormer: Searching Vision Transformers for Multi-task Learningcited by 0
  25. 2023Virtual Accessory Try-On via Keypoint Hallucinationcited by 0
  26. 2024AutoTask: Task Aware Multi-Faceted Single Model for Multi-Task Ads Relevancecited by 0
  27. 2024A dual-task mutual learning framework for predicting post-thrombectomy cerebral hemorrhagecited by 0
  28. 2024DLM-VMTL:A Double Layer Mapper for heterogeneous data video Multi-task prompt learningcited by 0
  29. 2025Multi-task Learning For Joint Action and Gesture Recognitioncited by 0
  30. 2025Vision Generalist Model: A Surveycited by 0
  31. 2025A Survey on Deep Multi-Task Learning in Connected Autonomous Vehiclescited by 0
  32. 2025Efficient Inter-Task Attention for Multitask Transformer Modelscited by 0
  33. 2025Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Modelscited by 0
  34. 2025Agentic Meta-Orchestrator for Multi-task Copilotscited by 0
  35. 2026Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localizationcited by 0