# Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

arXiv: https://arxiv.org/abs/2009.08445 · HTML: https://mlsanity.com/paper/2009.08445/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 51)

- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Revisiting Self-Training for Few-Shot Learning of Language Model](https://mlsanity.com/paper/2110.01256/) (2021)
- [Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning](https://mlsanity.com/paper/2105.14099/) (2021)
- [Contrastive Learning for Prompt-Based Few-Shot Language Learners](https://mlsanity.com/paper/2205.01308/) (2022)
- [Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization](https://mlsanity.com/paper/2210.09175/) (2022)
- [Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning](https://mlsanity.com/paper/2110.09446/) (2021)
- [Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models](https://mlsanity.com/paper/2210.14803/) (2022)
- [Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data](https://mlsanity.com/paper/2302.00674/) (2023)
- [Improving Both Domain Robustness and Domain Adaptability in Machine Translation](https://mlsanity.com/paper/2112.08288/) (2021)
- [FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue](https://mlsanity.com/paper/2205.06262/) (2022)
- [TAPE: Assessing Few-shot Russian Language Understanding](https://mlsanity.com/paper/2210.12813/) (2022)
- [kFolden: k-Fold Ensemble for Out-Of-Distribution Detection](https://mlsanity.com/paper/2108.12731/) (2021)
- [Knowledge-Aware Meta-learning for Low-Resource Text Classification](https://mlsanity.com/paper/2109.04707/) (2021)
- [Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP](https://mlsanity.com/paper/2111.01322/) (2021)
- [Continued Pretraining for Better Zero- and Few-Shot Promptability](https://mlsanity.com/paper/2210.10258/) (2022)
- [CLUES: Few-Shot Learning Evaluation in Natural Language Understanding](https://mlsanity.com/paper/2111.02570/) (2021)
- [ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback](https://mlsanity.com/paper/2107.14035/) (2021)
- [Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation](https://mlsanity.com/paper/2201.11576/) (2022)
- [FactMix: Using a Few Labeled In-domain Examples to Generalize to Cross-domain Named Entity Recognition](https://mlsanity.com/paper/2208.11464/) (2022)
- [Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta review](https://mlsanity.com/paper/2111.12146/) (2021)
- [Embedding Hallucination for Few-Shot Language Fine-tuning](https://mlsanity.com/paper/2205.01307/) (2022)
- [Meta-XNLG: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation](https://mlsanity.com/paper/2203.10250/) (2022)
- [On the cross-lingual transferability of multilingual prototypical models across NLU tasks](https://mlsanity.com/paper/2207.09157/) (2022)

## Abstract (arXiv)

Self-supervised pre-training of transformer models has revolutionized NLP applications. Such pre-training with language modeling objectives provides a useful initial point for parameters that generalize well to new tasks with fine-tuning. However, fine-tuning is still data inefficient -- when there are few labeled examples, accuracy can be low. Data efficiency can be improved by optimizing pre-training directly for future fine-tuning with few examples; this can be treated as a meta-learning problem. However, standard meta-learning techniques require many training tasks in order to generalize; unfortunately, finding a diverse set of such supervised tasks is usually difficult. This paper proposes a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text. This is achieved using a cloze-style objective, but creating separate multi-class classification tasks by gathering tokens-to-be blanked from among only a handful of vocabulary terms. This yields as many unique meta-training tasks as the number of subsets of vocabulary terms. We meta-train a transformer model on this distribution of tasks using a recent meta-learning framework. On 17 NLP tasks, we show that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning. Furthermore, we show how the self-supervised tasks can be combined with supervised tasks for meta-learning, providing substantial accuracy gains over previous supervised meta-learning.