# An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

arXiv: https://arxiv.org/abs/1902.10547 · HTML: https://mlsanity.com/paper/1902.10547/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2019.

## Built on by (most cited first, 30 of 39)

- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2106.06047/) (2021)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)
- [The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks](https://mlsanity.com/paper/2310.15469/) (2023)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)
- [INTERN: A New Learning Paradigm Towards General Vision](https://mlsanity.com/paper/2111.08687/) (2021)
- [Adaptive Transfer Learning on Graph Neural Networks](https://mlsanity.com/paper/2107.08765/) (2021)
- [AdaPrompt: Adaptive Model Training for Prompt-based NLP](https://mlsanity.com/paper/2202.04824/) (2022)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning](https://mlsanity.com/paper/2311.11551/) (2023)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning](https://mlsanity.com/paper/2109.05941/) (2021)
- [Diverse Pretrained Context Encodings Improve Document Translation](https://mlsanity.com/paper/2106.03717/) (2021)
- [Zero-Shot Learning for Requirements Classification: An Exploratory Study](https://mlsanity.com/paper/2302.04723/) (2023)
- [Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models](https://mlsanity.com/paper/2604.10079/) (2026)
- [Position-based Prompting for Health Outcome Generation](https://mlsanity.com/paper/2204.03489/) (2022)
- [LLMs Could Autonomously Learn Without External Supervision](https://mlsanity.com/paper/2406.00606/) (2024)
- [Understanding Knowledge Transferability for Transfer Learning: A Survey](https://mlsanity.com/paper/2507.03175/) (2025)
- [Cross-lingual Transferring of Pre-trained Contextualized Language Models](https://mlsanity.com/paper/2107.12627/) (2021)
- [Programming with Neural Surrogates of Programs](https://mlsanity.com/paper/2112.06148/) (2021)
- [Towards Automated Real-time Evaluation in Text-based Counseling](https://mlsanity.com/paper/2203.03442/) (2022)
- [AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model](https://mlsanity.com/paper/2211.11363/) (2022)
- [Analysis of Climate Campaigns on Social Media using Bayesian Model Averaging](https://mlsanity.com/paper/2305.06174/) (2023)
- [Leveraging Large Language Models for Entity Matching](https://mlsanity.com/paper/2405.20624/) (2024)
- [DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis](https://mlsanity.com/paper/2504.11082/) (2025)
- [Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning](https://mlsanity.com/paper/2212.03760/) (2022)
- [A Deeper (Autoregressive) Approach to Non-Convergent Discourse Parsing](https://mlsanity.com/paper/2305.12510/) (2023)

## Abstract (arXiv)

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.