# When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification

arXiv: https://arxiv.org/abs/1908.08788 · HTML: https://mlsanity.com/paper/1908.08788/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2019.

## Built on by (most cited first, 13 of 13)

- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2108.13161/) (2021)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Learning from Few Examples: A Summary of Approaches to Few-Shot Learning](https://mlsanity.com/paper/2203.04291/) (2022)
- [Ontology-enhanced Prompt-tuning for Few-shot Learning](https://mlsanity.com/paper/2201.11332/) (2022)
- [Improving Large-scale Language Models and Resources for Filipino](https://mlsanity.com/paper/2111.06053/) (2021)
- [DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population](https://mlsanity.com/paper/2201.03335/) (2022)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classification](https://mlsanity.com/paper/2401.04025/) (2024)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [MetaLDC: Meta Learning of Low-Dimensional Computing Classifiers for Fast On-Device Adaption](https://mlsanity.com/paper/2302.12347/) (2023)
- [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers](https://mlsanity.com/paper/2306.13804/) (2023)
- [Unsupervised Meta Learning for One Shot Title Compression in Voice Commerce](https://mlsanity.com/paper/2102.10760/) (2021)

## Abstract (arXiv)

Text classification tends to be difficult when data are deficient or when it is required to adapt to unseen classes. In such challenging scenarios, recent studies have often used meta-learning to simulate the few-shot task, thus negating implicit common linguistic features across tasks. This paper addresses such problems using meta-learning and unsupervised language models. Our approach is based on the insight that having a good generalization from a few examples relies on both a generic model initialization and an effective strategy for adapting this model to newly arising tasks. We show that our approach is not only simple but also produces a state-of-the-art performance on a well-studied sentiment classification dataset. It can thus be further suggested that pretraining could be a promising solution for few-shot learning of many other NLP tasks. The code and the dataset to replicate the experiments are made available at https://github.com/zxlzr/FewShotNLP.