# Pre-training Text Representations as Meta Learning

arXiv: https://arxiv.org/abs/2004.05568 · HTML: https://mlsanity.com/paper/2004.05568/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [Cluster & Tune: Boost Cold Start Performance in Text Classification](https://mlsanity.com/paper/2203.10581/) (2022)
- [Adversarial Self-Attention for Language Understanding](https://mlsanity.com/paper/2206.12608/) (2022)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning](https://mlsanity.com/paper/2210.10293/) (2022)
- [Cross-corpus Readability Compatibility Assessment for English Texts](https://mlsanity.com/paper/2306.09704/) (2023)

## Abstract (arXiv)

Pre-training text representations has recently been shown to significantly improve the state-of-the-art in many natural language processing tasks. The central goal of pre-training is to learn text representations that are useful for subsequent tasks. However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling. In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps. The standard multi-task learning objective adopted in BERT is a special case of our learning algorithm where the depth of meta-train is zero. We study the problem in two settings: unsupervised pre-training and supervised pre-training with different pre-training objects to verify the generality of our approach.Experimental results show that our algorithm brings improvements and learns better initializations for a variety of downstream tasks.