mlsanityData to 14 Oct 2026

Vision Transformer Adapters for Generalizable Multitask Learning

arXiv 2308.12372 ↗First posted 23 Aug 2023cs.CV cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We introduce the first multitasking vision transformer adapters that learn generalizable task affinities which can be applied to novel tasks and domains. Integrated into an off-the-shelf vision transformer backbone, our adapters can simultaneously solve multiple dense vision tasks in a parameter-efficient manner, unlike existing multitasking transformers that are parametrically expensive. In contrast to concurrent methods, we do not require retraining or fine-tuning whenever a new task or domain is added. We introduce a task-adapted attention mechanism within our adapter framework that combines gradient-based task similarities with attention-based ones. The learned task affinities generalize to the following settings: zero-shot task transfer, unsupervised domain adaptation, and generalization without fine-tuning to novel domains. We demonstrate that our approach outperforms not only the existing convolutional neural network-based multitasking methods but also the vision transformer-based ones. Our project page is at urlhttps://ivrl.github.io/VTAGML.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 60 of its 74 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

60 of 74 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2016Layer Normalizationcited by 5,669
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  11. 2017Decoupled Weight Decay Regularizationcited by 1,831
  12. 2021Vision Transformers for Dense Predictioncited by 1,803
38 more
  1. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  2. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  3. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  5. 2020Pre-Trained Image Processing Transformercited by 893
  6. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  8. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  9. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  10. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  11. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  13. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  14. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  15. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  16. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  17. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  18. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  19. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  20. 2020Virtual KITTI 2cited by 353
  21. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  22. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  23. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  24. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  25. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  26. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  27. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  28. 2022Multi-class Token Transformer for Weakly Supervised Semantic Segmentationcited by 138
  29. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  30. 2020General Multi-label Image Classification with Transformerscited by 127
  31. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 127
  32. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  33. 2020Mirror Descent Policy Optimizationcited by 106
  34. 2020Robust Learning Through Cross-Task Consistencycited by 105
  35. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  36. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  37. 2022TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentationcited by 82
  38. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 6; 2026: 5.

14 papers.

Sort:

By area:

  1. 2023PolyMaX: General Dense Prediction with Mask Transformercited by 14
  2. 2023Multi-task Learning with 3D-Aware Regularizationcited by 8
  3. 2025PAVE: Patching and Adapting Video Large Language Modelscited by 4
  4. 2025Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Taskscited by 2
  5. 20253D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understandingcited by 2
  6. 2026Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentationcited by 2
  7. 2025Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Expertscited by 1
  8. 2024Efficient Multi-domain Text Recognition Deep Neural Network Parameterization with Residual Adapterscited by 0
  9. 2025Efficient Domain-Adaptive Multi-Task Dense Prediction with Vision Foundation Modelscited by 0
  10. 2025MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapterscited by 0
  11. 2026Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Modelscited by 0
  12. 2026Hidden-Shot: Towards One-Shot Task Generalization for Low-Level Vision Generalist Modelscited by 0
2 more
  1. 2026Task-State Adaptation with Prototype Memory for Multi-Task Dense Predictioncited by 0
  2. 2026SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learningcited by 0