# Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes

arXiv: https://arxiv.org/abs/2211.13638 · HTML: https://mlsanity.com/paper/2211.13638/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 57 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Robust Classification with Convolutional Prototype Learning](https://mlsanity.com/paper/1805.03438/) (2018)
- [Infinite Mixture Prototypes for Few-Shot Learning](https://mlsanity.com/paper/1902.04552/) (2019)
- [Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification](https://mlsanity.com/paper/1608.04363/) (2016)
- [Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning](https://mlsanity.com/paper/2106.09226/) (2021)
- [The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification](https://mlsanity.com/paper/1503.01161/) (2015)
- [Interpretable and Steerable Sequence Learning via Prototypes](https://mlsanity.com/paper/1907.09728/) (2019)
- [GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective](https://mlsanity.com/paper/2211.08073/) (2022)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better](https://mlsanity.com/paper/2202.12024/) (2022)

## Built on by (most cited first, 10 of 10)

- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [Continual Learning on Dynamic Graphs via Parameter Isolation](https://mlsanity.com/paper/2305.13825/) (2023)
- [Prototypical Reward Network for Data-Efficient RLHF](https://mlsanity.com/paper/2406.06606/) (2024)
- [TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification](https://mlsanity.com/paper/2309.11845/) (2023)
- [Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts](https://mlsanity.com/paper/2502.08825/) (2025)
- [Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity](https://mlsanity.com/paper/2403.09428/) (2024)
- [ProtoTS: Learning Hierarchical Prototypes for Explainable Time Series Forecasting](https://mlsanity.com/paper/2509.23159/) (2025)
- [Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)](https://mlsanity.com/paper/2409.16594/) (2024)
- [ProtoBERT-LoRA: Parameter-Efficient Prototypical Finetuning for Immunotherapy Study Identification](https://mlsanity.com/paper/2503.20179/) (2025)
- [Causal Connections: Leveraging Multilingual Fine-Tuning for Financial QA@FinCausal 2026](https://mlsanity.com/paper/2606.27446/) (2026)

## Abstract (arXiv)

In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model's inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.