mlsanityData to 7 Oct 2026

Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning

arXiv 2312.03248 ↗First posted 6 Dec 2023cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon C-Poly that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques. Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

28 of 44 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
16 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  3. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  4. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  5. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  6. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  7. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  8. 2017The Consciousness Priorcited by 105
  9. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  10. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  11. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  12. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  13. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  14. 2022Combining Modular Skills in Multitask Learningcited by 27
  15. 2015Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domaincited by 19
  16. 2021Neural Network Module Decomposition and Recompositioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 0; 2026: 3.

8 papers.

  1. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  2. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  3. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  4. 2024Simple Drop-in LoRA Conditioning on Attention Layers Will Improve Your Diffusion Modelcited by 6
  5. 2024OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergycited by 2
  6. 2026REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modelingcited by 1
  7. 2026PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Promptscited by 0
  8. 2026MTL-FNO: A Lightweight Multi-Task Fourier Neural Operator for Sparse Field Reconstructioncited by 0