# Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

arXiv: https://arxiv.org/abs/1911.03863 · HTML: https://mlsanity.com/paper/1911.03863/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 30 of 57)

- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [Advances and Challenges in Meta-Learning: A Technical Review](https://mlsanity.com/paper/2307.04722/) (2023)
- [DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer](https://mlsanity.com/paper/2312.03724/) (2023)
- [When Brain-inspired AI Meets AGI](https://mlsanity.com/paper/2303.15935/) (2023)
- [ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning](https://mlsanity.com/paper/2212.01378/) (2022)
- [Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning](https://mlsanity.com/paper/2104.08808/) (2021)
- [Controllable Semantic Parsing via Retrieval Augmentation](https://mlsanity.com/paper/2110.08458/) (2021)
- [Joint Prompt Optimization of Stacked LLMs using Variational Inference](https://mlsanity.com/paper/2306.12509/) (2023)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning](https://mlsanity.com/paper/2203.09137/) (2022)
- [Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning](https://mlsanity.com/paper/2105.14099/) (2021)
- [Imagination-Augmented Natural Language Understanding](https://mlsanity.com/paper/2204.08535/) (2022)
- [Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind](https://mlsanity.com/paper/2211.04684/) (2022)
- [ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classification](https://mlsanity.com/paper/2305.09269/) (2023)
- [Few Shot Dialogue State Tracking using Meta-learning](https://mlsanity.com/paper/2101.06779/) (2021)
- [Example-based Hypernetworks for Out-of-Distribution Generalization](https://mlsanity.com/paper/2203.14276/) (2022)
- [Multilingual and cross-lingual document classification: A meta-learning approach](https://mlsanity.com/paper/2101.11302/) (2021)
- [Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning](https://mlsanity.com/paper/2110.09446/) (2021)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives](https://mlsanity.com/paper/2411.05818/) (2024)
- [TAPE: Assessing Few-shot Russian Language Understanding](https://mlsanity.com/paper/2210.12813/) (2022)
- [Minimax and Neyman-Pearson Meta-Learning for Outlier Languages](https://mlsanity.com/paper/2106.01051/) (2021)
- [Knowledge-Aware Meta-learning for Low-Resource Text Classification](https://mlsanity.com/paper/2109.04707/) (2021)
- [Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP](https://mlsanity.com/paper/2111.01322/) (2021)

## Abstract (arXiv)

Self-supervised pre-training of transformer models has shown enormous success in improving performance on a number of downstream tasks. However, fine-tuning on a new task still requires large amounts of task-specific labelled data to achieve good performance. We consider this problem of learning to generalize to new tasks with few examples as a meta-learning problem. While meta-learning has shown tremendous progress in recent years, its application is still limited to simulated problems or problems with limited diversity across tasks. We develop a novel method, LEOPARD, which enables optimization-based meta-learning across tasks with different number of classes, and evaluate different methods on generalization to diverse NLP classification tasks. LEOPARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label. Across 17 NLP tasks, including diverse domains of entity typing, natural language inference, sentiment analysis, and several other text classification tasks, we show that LEOPARD learns better initial parameters for few-shot learning than self-supervised pre-training or multi-task training, outperforming many strong baselines, for example, yielding 14.5% average relative gain in accuracy on unseen tasks with only 4 examples per label.