mlsanityData to 7 Oct 2026

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

arXiv 1902.10547 ↗First posted 27 Feb 2019cs.CL cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 7; 2023: 9; 2024: 5; 2025: 2; 2026: 3.

39 papers.

Sort:

By area:

  1. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  2. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  3. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  4. 2022Transferability in Deep Learning: A Surveycited by 57
  5. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
  6. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  7. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35
  8. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  9. 2021INTERN: A New Learning Paradigm Towards General Visioncited by 33
  10. 2021Adaptive Transfer Learning on Graph Neural Networkscited by 31
  11. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  12. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
27 more
  1. 2023Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learningcited by 16
  2. 2021Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generationcited by 14
  3. 2021Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learningcited by 14
  4. 2021Diverse Pretrained Context Encodings Improve Document Translationcited by 13
  5. 2023Zero-Shot Learning for Requirements Classification: An Exploratory Studycited by 9
  6. 2022Position-based Prompting for Health Outcome Generationcited by 7
  7. 2026Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Modelscited by 7
  8. 2024LLMs Could Autonomously Learn Without External Supervisioncited by 6
  9. 2025Understanding Knowledge Transferability for Transfer Learning: A Surveycited by 4
  10. 2021Cross-lingual Transferring of Pre-trained Contextualized Language Modelscited by 3
  11. 2021Programming with Neural Surrogates of Programscited by 3
  12. 2022Towards Automated Real-time Evaluation in Text-based Counselingcited by 3
  13. 2022AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Modelcited by 3
  14. 2023Analysis of Climate Campaigns on Social Media using Bayesian Model Averagingcited by 3
  15. 2024Leveraging Large Language Models for Entity Matchingcited by 3
  16. 2025DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysiscited by 3
  17. 2022Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learningcited by 2
  18. 2023A Deeper (Autoregressive) Approach to Non-Convergent Discourse Parsingcited by 2
  19. 2024Transfer Learning with Clinical Concept Embeddings from Large Language Modelscited by 2
  20. 2021Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translationcited by 1
  21. 2023Leveraging Twitter Data for Sentiment Analysis of Transit User Feedback: An NLP Frameworkcited by 1
  22. 2021One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Datacited by 0
  23. 2022A Semi-Supervised Deep Clustering Pipeline for Mining Intentions From Textscited by 0
  24. 2023DomainInv: Domain Invariant Fine Tuning and Adversarial Label Correction For QA Domain Adaptationcited by 0
  25. 2024CVE-LLM : Automatic vulnerability evaluation in medical device industry using large language modelscited by 0
  26. 2026MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptationcited by 0
  27. 2026Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysiscited by 0