# K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning

arXiv: https://arxiv.org/abs/1810.10703 · HTML: https://mlsanity.com/paper/1810.10703/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2018.

## Built on by (most cited first, 30 of 34)

- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets](https://mlsanity.com/paper/2208.07463/) (2022)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [Fine-tuning Image Transformers using Learnable Memory](https://mlsanity.com/paper/2203.15243/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation](https://mlsanity.com/paper/2205.03076/) (2022)
- [FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification](https://mlsanity.com/paper/2206.08671/) (2022)
- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [Multi-Scale Aligned Distillation for Low-Resolution Detection](https://mlsanity.com/paper/2109.06875/) (2021)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation](https://mlsanity.com/paper/2303.17051/) (2023)
- [See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition](https://mlsanity.com/paper/2407.05417/) (2024)
- [The Expressive Power of Tuning Only the Normalization Layers](https://mlsanity.com/paper/2302.07937/) (2023)
- [On the Efficacy of Differentially Private Few-shot Image Classification](https://mlsanity.com/paper/2302.01190/) (2023)
- [ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection](https://mlsanity.com/paper/2312.14227/) (2023)
- [Universal Deep Image Compression via Content-Adaptive Optimization with Adapters](https://mlsanity.com/paper/2211.00918/) (2022)
- [Task Generalization with Stability Guarantees via Elastic Dynamical System Motion Policies](https://mlsanity.com/paper/2309.01884/) (2023)
- [BasisNet: Two-stage Model Synthesis for Efficient Inference](https://mlsanity.com/paper/2105.03014/) (2021)
- [Rethinking Hard-Parameter Sharing in Multi-Domain Learning](https://mlsanity.com/paper/2107.11359/) (2021)
- [MobileTL: On-device Transfer Learning with Inverted Residual Blocks](https://mlsanity.com/paper/2212.03246/) (2022)
- [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](https://mlsanity.com/paper/2407.10734/) (2024)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [On-device Self-supervised Learning of Visual Perception Tasks aboard Hardware-limited Nano-quadrotors](https://mlsanity.com/paper/2403.04071/) (2024)
- [Training on the Fly: On-device Self-supervised Learning aboard Nano-drones within 20 mW](https://mlsanity.com/paper/2408.03168/) (2024)
- [SpotPatch: Parameter-Efficient Transfer Learning for Mobile Object Detection](https://mlsanity.com/paper/2101.01260/) (2021)
- [Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks](https://mlsanity.com/paper/2107.10963/) (2021)
- [Hyperparameters in Score-Based Membership Inference Attacks](https://mlsanity.com/paper/2502.06374/) (2025)
- [AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments](https://mlsanity.com/paper/2410.08256/) (2024)
- [Simplifying Multi-Task Architectures Through Task-Specific Normalization](https://mlsanity.com/paper/2512.20420/) (2025)
- [Unidirectional Thin Adapter for Efficient Adaptation of Deep Neural Networks](https://mlsanity.com/paper/2203.10463/) (2022)

## Abstract (arXiv)

We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.