mlsanityData to 7 Oct 2026

AutoTaskFormer: Searching Vision Transformers for Multi-task Learning

arXiv 2304.08756 ↗First posted 18 Apr 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision Transformers have shown great performance in single tasks such as classification and segmentation. However, real-world problems are not isolated, which calls for vision transformers that can perform multiple tasks concurrently. Existing multi-task vision transformers are handcrafted and heavily rely on human expertise. In this work, we propose a novel one-shot neural architecture search framework, dubbed AutoTaskFormer (Automated Multi-Task Vision TransFormer), to automate this process. AutoTaskFormer not only identifies the weights to share across multiple tasks automatically, but also provides thousands of well-trained vision transformers with a wide range of parameters (e.g., number of heads and network depth) for deployment under various resource constraints. Experiments on both small-scale (2-task Cityscapes and 3-task NYUv2) and large-scale (16-task Taskonomy) datasets show that AutoTaskFormer outperforms state-of-the-art handcrafted vision transformers in multi-task learning. The entire code and models will be open-sourced.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 39 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

33 of 39 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  9. 2016Pyramid Scene Parsing Networkcited by 2,472
  10. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  11. 2020Pre-Trained Image Processing Transformercited by 891
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
21 more
  1. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  2. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  3. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  5. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  6. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  7. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  8. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  9. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  10. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  11. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  12. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  13. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  14. 2020Learning to Branch for Multi-Task Learningcited by 114
  15. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  16. 2020MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learningcited by 50
  17. 2022MulT: An End-to-End Multitask Learning Transformercited by 47
  18. 2021Searching the Search Space of Vision Transformercited by 39
  19. 2019Deep Elastic Networks with Model Selection for Multi-Task Learningcited by 20
  20. 2021AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learningcited by 17
  21. 2021Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentationcited by 9

Built on by

Not cited yet.