# Evolution of transfer learning in natural language processing

arXiv: https://arxiv.org/abs/1910.07370 · HTML: https://mlsanity.com/paper/1910.07370/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [From Artificially Real to Real: Leveraging Pseudo Data from Large Language Models for Low-Resource Molecule Discovery](https://mlsanity.com/paper/2309.05203/) (2023)
- [Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting](https://mlsanity.com/paper/2406.10327/) (2024)
- [Deep Sequence Models for Text Classification Tasks](https://mlsanity.com/paper/2207.08880/) (2022)
- [Leveraging Domain Knowledge for Inclusive and Bias-aware Humanitarian Response Entry Classification](https://mlsanity.com/paper/2305.16756/) (2023)
- [AgriLLM: Harnessing Transformers for Farmer Queries](https://mlsanity.com/paper/2407.04721/) (2024)
- [Contextualized Embeddings based Convolutional Neural Networks for Duplicate Question Identification](https://mlsanity.com/paper/2109.01560/) (2021)
- [Knothe-Rosenblatt transport for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2110.02716/) (2021)
- [Natural Language Processing Through Transfer Learning: A Case Study on Sentiment Analysis](https://mlsanity.com/paper/2311.16965/) (2023)

## Abstract (arXiv)

In this paper, we present a study of the recent advancements which have helped bring Transfer Learning to NLP through the use of semi-supervised training. We discuss cutting-edge methods and architectures such as BERT, GPT, ELMo, ULMFit among others. Classically, tasks in natural language processing have been performed through rule-based and statistical methodologies. However, owing to the vast nature of natural languages these methods do not generalise well and failed to learn the nuances of language. Thus machine learning algorithms such as Naive Bayes and decision trees coupled with traditional models such as Bag-of-Words and N-grams were used to usurp this problem. Eventually, with the advent of advanced recurrent neural network architectures such as the LSTM, we were able to achieve state-of-the-art performance in several natural language processing tasks such as text classification and machine translation. We talk about how Transfer Learning has brought about the well-known ImageNet moment for NLP. Several advanced architectures such as the Transformer and its variants have allowed practitioners to leverage knowledge gained from unrelated task to drastically fasten convergence and provide better performance on the target task. This survey represents an effort at providing a succinct yet complete understanding of the recent advances in natural language processing using deep learning in with a special focus on detailing transfer learning and its potential advantages.