mlsanityData to 14 Oct 2026

Prompt Guided Transformer for Multi-Task Dense Prediction

arXiv 2307.15362 ↗First posted 28 Jul 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Task-conditional architecture offers advantage in parameter efficiency but falls short in performance compared to state-of-the-art multi-decoder methods. How to trade off performance and model parameters is an important and difficult problem. In this paper, we introduce a simple and lightweight task-conditional model called Prompt Guided Transformer (PGT) to optimize this challenge. Our approach designs a Prompt-conditioned Transformer block, which incorporates task-specific prompts in the self-attention mechanism to achieve global dependency modeling and parameter-efficient feature adaptation across multiple tasks. This block is integrated into both the shared encoder and decoder, enhancing the capture of intra- and inter-task features. Moreover, we design a lightweight decoder to further reduce parameter usage, which accounts for only 2.7% of the total model parameters. Extensive experiments on two multi-task dense prediction benchmarks, PASCAL-Context and NYUD-v2, demonstrate that our approach achieves state-of-the-art results among task-conditional methods while using fewer parameters, and maintains a significant balance between performance and parameter size.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 58 of its 87 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

58 of 87 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  5. 2021Vision Transformers for Dense Predictioncited by 1,803
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  9. 2021Learning to Prompt for Continual Learningcited by 944
  10. 2020Pre-Trained Image Processing Transformercited by 893
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  13. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  14. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  15. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  16. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  17. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  18. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  19. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  20. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  21. 2018Video Action Transformer Networkcited by 316
  22. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  23. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  24. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  25. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  26. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 142
  27. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  28. 2020Learning to Branch for Multi-Task Learningcited by 115
  29. 2016Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classificationcited by 93
  30. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  31. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  32. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  33. 2016Deep Multi-task Representation Learning: A Tensor Factorisation Approachcited by 69
  34. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  35. 2020Automated Search for Resource-Efficient Branched Multi-Task Networkscited by 51
  36. 2020Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interferencecited by 47
  37. 2022PIVOT: Prompting for Video Continual Learningcited by 45
  38. 2019Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernelscited by 42

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 6; 2026: 3.

13 papers.

Sort:

By area:

  1. 2023FedHCA2: Towards Hetero-Client Federated Multi-Task Learningcited by 17
  2. 2025A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detectioncited by 9
  3. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 7
  4. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 5
  5. 2024YOLO-MED : Multi-Task Interaction Network for Biomedical Imagescited by 3
  6. 2024Implicit and Explicit Language Guidance for Diffusion-based Visual Perceptioncited by 3
  7. 2025From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarioscited by 3
  8. 2025SAMwave: Wavelet-Driven Feature Enrichment for Effective Adaptation of Segment Anything Modelcited by 3
  9. 2025StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasetscited by 2
  10. 2025Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Expertscited by 1
  11. 2026Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidancecited by 0
  12. 2026Unified Video Dense Prediction from Disjoint Datacited by 0
1 more
  1. 2026Task-State Adaptation with Prototype Memory for Multi-Task Dense Predictioncited by 0