mlsanityData to 7 Oct 2026

Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning

arXiv 2402.15017 ↗First posted 22 Feb 2024cs.LG cs.AI cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Foundation models have emerged as a powerful tool for many AI problems. Despite the tremendous success of foundation models, effective adaptation to new tasks, particularly those with limited labels, remains an open question and lacks theoretical understanding. An emerging solution with recent success in vision and NLP involves finetuning a foundation model on a selection of relevant tasks, before its adaptation to a target task with limited labeled samples. In this paper, we study the theoretical justification of this multitask finetuning approach. Our theoretical analysis reveals that with a diverse set of related tasks, this multitask finetuning leads to reduced error in the target task, in comparison to directly adapting the same pretrained model. We quantify the relationship between finetuning tasks and target tasks by diversity and consistency metrics, and further propose a practical task selection algorithm. We substantiate our theoretical claims with extensive empirical evidence. Further, we present results affirming our task selection algorithm adeptly chooses related finetuning tasks, providing advantages to the model performance on target tasks. We believe our study shed new light on the effective adaptation of foundation models to new tasks that lack abundant labels. Our code is available at https://github.com/OliverXUZY/Foudation-Model_Multitask.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 48 of its 104 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

48 of 104 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
36 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  13. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  14. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  15. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  16. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  17. 2021True Few-Shot Learning with Language Modelscited by 343
  18. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  19. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  20. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  21. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  22. 2018An efficient framework for learning sentence representationscited by 197
  23. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  24. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  25. 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
  26. 2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
  27. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  28. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  29. 2023A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learningcited by 22
  30. 2023When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysiscited by 21
  31. 2020Functional Regularization for Representation Learning: A Unified Theoretical Perspectivecited by 18
  32. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  33. 2023Domain Generalization via Nuclear Norm Regularizationcited by 12
  34. 2022Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifierscited by 7
  35. 2023Geometry-Aware Adaptation for Pretrained Modelscited by 5
  36. 2022Blessing of Class Diversity in Pre-trainingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 14; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  2. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  3. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  4. 2024Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspectivecited by 24
  5. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  6. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  7. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  8. 2024Bayesian-guided Label Mapping for Visual Reprogrammingcited by 12
  9. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  10. 2025Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolationcited by 11
  11. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  12. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 9
19 more
  1. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  2. 2025HuggingGraph: Understanding the Supply Chain of LLM Ecosystemcited by 8
  3. 2025Attribute-based Visual Reprogramming for Vision-Language Modelscited by 7
  4. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  5. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  6. 2025Neural Algorithmic Reasoning for Hypergraphs with Looped Transformerscited by 3
  7. 2026ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplificationcited by 3
  8. 2024Task Addition in Multi-Task Learning by Geometrical Alignmentcited by 2
  9. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2
  10. 2024Variational Model Merging for Pareto Front Estimation in Multitask Finetuningcited by 1
  11. 2025HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuningcited by 1
  12. 2025Scaling Laws for Geospatial Foundation Models: A case study on PhilEO Benchcited by 1
  13. 2024MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Datacited by 0
  14. 2025Few-Shot Learning for Industrial Time Series: A Comparative Analysis Using the Example of Screw-Fastening Process Monitoringcited by 0
  15. 2025Can Language Models Compose Skills In-Context?cited by 0
  16. 2026Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViTcited by 0
  17. 2026BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogrammingcited by 0
  18. 2026The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classificationcited by 0
  19. 2026Bridging the Synthetic-to-Real Gap for Few-Shot Cryo-ET Classificationcited by 0