mlsanityData to 7 Oct 2026

Projected Subnetworks Scale Adaptation

arXiv 2301.11487 ↗First posted 27 Jan 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large models support great zero-shot and few-shot capabilities. However, updating these models on new tasks can break performance on previous seen tasks and their zero/few-shot unseen tasks. Our work explores how to update zero/few-shot learners such that they can maintain performance on seen/unseen tasks of previous tasks as well as new tasks. By manipulating the parameter updates of a gradient-based meta learner as the projected task-specific subnetworks, we show improvements for large models to retain seen and zero/few shot task performance in online settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

36 of 46 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
24 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  5. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  6. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  8. 2015Neural Module Networkscited by 375
  9. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  10. 2021Gradient Projection Memory for Continual Learningcited by 338
  11. 2019Model Fusion via Optimal Transportcited by 217
  12. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  13. 2022Patching open-vocabulary models by interpolating weightscited by 187
  14. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  15. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  16. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  17. 2021Similarity and Matching of Neural Network Representationscited by 77
  18. 2020RATT: Recurrent Attention to Transient Tasks for Continual Image Captioningcited by 32
  19. 2021What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoningcited by 22
  20. 2022Learn2Weight: Parameter Adaptation against Similar-domain Adversarial Attackscited by 5
  21. 2022Backdoors Stuck At The Frontdoor: Multi-Agent Backdoor Attacks That Backfirecited by 4
  22. 2022Low-Loss Subspace Compression for Clean Gains against Multi-Agent Backdoor Attackscited by 4
  23. 2022Interpolating Compressed Parameter Subspacescited by 3
  24. 2022Multiple Modes for Continual Learningcited by 2

Built on by

Not cited yet.