# Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

arXiv: https://arxiv.org/abs/1908.10423 · HTML: https://mlsanity.com/paper/1908.10423/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2019.

## Built on by (most cited first, 30 of 48)

- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learning](https://mlsanity.com/paper/2308.00788/) (2023)
- [Controllable Semantic Parsing via Retrieval Augmentation](https://mlsanity.com/paper/2110.08458/) (2021)
- [Meta Learning for Natural Language Processing: A Survey](https://mlsanity.com/paper/2205.01500/) (2022)
- [MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning](https://mlsanity.com/paper/2104.07908/) (2021)
- [ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning](https://mlsanity.com/paper/2105.12995/) (2021)
- [Few Shot Dialogue State Tracking using Meta-learning](https://mlsanity.com/paper/2101.06779/) (2021)
- [CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models](https://mlsanity.com/paper/2302.04030/) (2023)
- [On sensitivity of meta-learning to support data](https://mlsanity.com/paper/2110.13953/) (2021)
- [Multilingual and cross-lingual document classification: A meta-learning approach](https://mlsanity.com/paper/2101.11302/) (2021)
- [Meta-Transfer Learning for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2102.09397/) (2021)
- [Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD](https://mlsanity.com/paper/2109.07497/) (2021)
- [Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation](https://mlsanity.com/paper/2207.03509/) (2022)
- [Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings](https://mlsanity.com/paper/2410.13153/) (2024)
- [Meta-Learning for Effective Multi-task and Multilingual Modelling](https://mlsanity.com/paper/2101.10368/) (2021)
- [Improving Both Domain Robustness and Domain Adaptability in Machine Translation](https://mlsanity.com/paper/2112.08288/) (2021)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)
- [X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering](https://mlsanity.com/paper/2104.09696/) (2021)
- [Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP](https://mlsanity.com/paper/2111.01322/) (2021)
- [Continued Pretraining for Better Zero- and Few-Shot Promptability](https://mlsanity.com/paper/2210.10258/) (2022)
- [CLUES: Few-Shot Learning Evaluation in Natural Language Understanding](https://mlsanity.com/paper/2111.02570/) (2021)
- [Learning a Better Initialization for Soft Prompts via Meta-Learning](https://mlsanity.com/paper/2205.12471/) (2022)
- [EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching](https://mlsanity.com/paper/2210.12540/) (2022)
- [Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes](https://mlsanity.com/paper/2302.11709/) (2023)
- [Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding](https://mlsanity.com/paper/2306.11066/) (2023)
- [Few-shot Learning for Slot Tagging with Attentive Relational Network](https://mlsanity.com/paper/2103.02333/) (2021)
- [ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation](https://mlsanity.com/paper/2405.11912/) (2024)
- [On the cross-lingual transferability of multilingual prototypical models across NLU tasks](https://mlsanity.com/paper/2207.09157/) (2022)

## Abstract (arXiv)

Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based meta-learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively.