# Modular Meta-Learning with Shrinkage

arXiv: https://arxiv.org/abs/1909.05557 · HTML: https://mlsanity.com/paper/1909.05557/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 30 of 31)

- [FedBABU: Towards Enhanced Representation for Federated Image Classification](https://mlsanity.com/paper/2106.06042/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [Generalizing to New Physical Systems via Context-Informed Dynamics Model](https://mlsanity.com/paper/2202.01889/) (2022)
- [Learning where to learn: Gradient sparsity in meta and continual learning](https://mlsanity.com/paper/2110.14402/) (2021)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Prediction](https://mlsanity.com/paper/2205.02708/) (2022)
- [A contrastive rule for meta-learning](https://mlsanity.com/paper/2104.01677/) (2021)
- [Continuous-Time Meta-Learning with Forward Mode Differentiation](https://mlsanity.com/paper/2203.01443/) (2022)
- [Physics-Informed Deep Learning For Traffic State Estimation: A Survey and the Outlook](https://mlsanity.com/paper/2303.02063/) (2023)
- [An Expectation-Maximization Perspective on Federated Learning](https://mlsanity.com/paper/2111.10192/) (2021)
- [Few-Shot Learning by Dimensionality Reduction in Gradient Space](https://mlsanity.com/paper/2206.03483/) (2022)
- [On the Generalization and Adaption Performance of Causal Models](https://mlsanity.com/paper/2206.04620/) (2022)
- [Stochastic Deep Networks with Linear Competing Units for Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2208.01573/) (2022)
- [Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification](https://mlsanity.com/paper/2206.09843/) (2022)
- [Breaking Neural Network Scaling Laws with Modularity](https://mlsanity.com/paper/2409.05780/) (2024)
- [From Multi-label Learning to Cross-Domain Transfer: A Model-Agnostic Approach](https://mlsanity.com/paper/2207.11742/) (2022)
- [Test like you Train in Implicit Deep Learning](https://mlsanity.com/paper/2305.15042/) (2023)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification](https://mlsanity.com/paper/2312.07338/) (2023)
- [Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition](https://mlsanity.com/paper/2106.04392/) (2021)
- [Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter](https://mlsanity.com/paper/2309.09443/) (2023)
- [GMTR: Graph Matching Transformers](https://mlsanity.com/paper/2311.08141/) (2023)
- [DarkShot: Lighting Dark Images with Low-Compute and High-Quality](https://mlsanity.com/paper/2312.16805/) (2023)
- [TRLS: A Time Series Representation Learning Framework via Spectrogram for Medical Signal Processing](https://mlsanity.com/paper/2401.05431/) (2024)
- [TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks](https://mlsanity.com/paper/2401.05432/) (2024)
- [T-Pixel2Mesh: Combining Global and Local Transformer for 3D Mesh Generation from a Single Image](https://mlsanity.com/paper/2403.13663/) (2024)
- [Strategic Arms with Side Communication Prevail Over Low-Regret MAB Algorithms](https://mlsanity.com/paper/2408.17101/) (2024)
- [Towards General Deepfake Detection with Dynamic Curriculum](https://mlsanity.com/paper/2410.11162/) (2024)
- [Attr-Int: A Simple and Effective Entity Alignment Framework for Heterogeneous Knowledge Graphs](https://mlsanity.com/paper/2410.13409/) (2024)

## Abstract (arXiv)

Many real-world problems, including multi-speaker text-to-speech synthesis, can greatly benefit from the ability to meta-learn large models with only a few task-specific components. Updating only these task-specific modules then allows the model to be adapted to low-data tasks for as many steps as necessary without risking overfitting. Unfortunately, existing meta-learning methods either do not scale to long adaptation or else rely on handcrafted task-specific architectures. Here, we propose a meta-learning approach that obviates the need for this often sub-optimal hand-selection. In particular, we develop general techniques based on Bayesian shrinkage to automatically discover and learn both task-specific and general reusable modules. Empirically, we demonstrate that our method discovers a small set of meaningful task-specific modules and outperforms existing meta-learning approaches in domains like few-shot text-to-speech that have little task data and long adaptation horizons. We also show that existing meta-learning methods including MAML, iMAML, and Reptile emerge as special cases of our method.