mlsanityData to 14 Oct 2026

Controllable Dynamic Multi-Task Architectures

arXiv 2203.14949 ↗First posted 28 Mar 2022cs.CV cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Multi-task learning commonly encounters competition for resources among tasks, specifically when model capacity is limited. This challenge motivates models which allow control over the relative importance of tasks and total compute cost during inference time. In this work, we propose such a controllable multi-task network that dynamically adjusts its architecture and weights to match the desired task preference as well as the resource constraints. In contrast to the existing dynamic multi-task approaches that adjust only the weights within a fixed architecture, our approach affords the flexibility to dynamically control the total computational cost and match the user-preferred task importance better. We propose a disentangled training of two hypernetworks, by exploiting task affinity and a novel branching regularized loss, to take input preferences and accordingly predict tree-structured models with adapted weights. Experiments on three multi-task benchmarks, namely PASCAL-Context, NYU-v2, and CIFAR-100, show the efficacy of our approach. Project page is available at https://www.nec-labs.com/~mas/DYMU.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 46 of its 60 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

46 of 60 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  6. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  7. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2016HyperNetworkscited by 1,027
  10. 2017A Survey on Multi-Task Learningcited by 840
  11. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  12. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
34 more
  1. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  3. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  4. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  5. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  6. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  7. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  8. 2021Dynamic Neural Networks: A Surveycited by 408
  9. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  10. 2016Dynamic Filter Networkscited by 319
  11. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  12. 2019Pareto Multi-Task Learningcited by 224
  13. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 202
  14. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  15. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  16. 2017Adaptive Neural Networks for Efficient Inferencecited by 163
  17. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 160
  18. 2017Convolutional Networks with Adaptive Inference Graphscited by 148
  19. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  20. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
  21. 2020Learning the Pareto Front with Hypernetworkscited by 125
  22. 2020Learning to Branch for Multi-Task Learningcited by 115
  23. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  24. 2021Dynamic Slimmable Networkcited by 83
  25. 2020Learning Dynamic Routing for Semantic Segmentationcited by 79
  26. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  27. 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
  28. 2019Deep Meta Functionals for Shape Representationcited by 53
  29. 2020Automated Search for Resource-Efficient Branched Multi-Task Networkscited by 51
  30. 2020Controllable Pareto Multi-Task Learningcited by 51
  31. 2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 35
  32. 2019Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Searchcited by 23
  33. 2017Changing Model Behavior at Test-Time Using Reinforcement Learningcited by 15
  34. 2019S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Searchcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 8; 2025: 3; 2026: 2.

21 papers.

Sort:

By area:

  1. 2022Multi-Task Learning with Multi-Query Transformer for Dense Predictioncited by 34
  2. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 18
  3. 2023Dynamic Neural Network for Multi-Task Learning Searching across Diverse Network Topologiescited by 9
  4. 2023Efficient Controllable Multi-Task Architecturescited by 7
  5. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 7
  6. 2024Multi-task Learning for Real-time Autonomous Driving Leveraging Task-adaptive Attention Generatorcited by 6
  7. 2025MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptationcited by 5
  8. 2023Tunable Convolutions with Parametric Multi-Loss Optimizationcited by 4
  9. 2024Neural Metamorphosiscited by 4
  10. 2023Bi-level Multi-objective Evolutionary Learning: A Case Study on Multi-task Graph Neural Topology Searchcited by 3
  11. 2024A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraintscited by 3
  12. 2024Towards Principled Task Grouping for Multi-Task Learningcited by 3
9 more
  1. 2024Federated Communication-Efficient Multi-Objective Optimizationcited by 3
  2. 2023Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removalcited by 2
  3. 2024Preference-Optimized Pareto Set Learning for Blackbox Optimizationcited by 2
  4. 2022GPPF: A General Perception Pre-training Framework via Sparsely Activated Multi-Task Learningcited by 1
  5. 2025Simplifying Multi-Task Architectures Through Task-Specific Normalizationcited by 1
  6. 2023JOINEDTrans: Prior Guided Multi-task Transformer for Joint Optic Disc/Cup Segmentation and Fovea Detectioncited by 0
  7. 2024Model Adaptation for Time Constrained Embodied Controlcited by 0
  8. 2026Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compressioncited by 0
  9. 2026Efficient Hessian-Free Methods for Multi-Objective Bilevel Optimization with Nonconvex Lower Levelcited by 0