# Paradigm Shift in Natural Language Processing

arXiv: https://arxiv.org/abs/2109.12575 · HTML: https://mlsanity.com/paper/2109.12575/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 78 of its 105 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 30 of 78)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Factual Probing Is [MASK]: Learning vs. Learning to Recall](https://mlsanity.com/paper/2104.05240/) (2021)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)

## Built on by (most cited first, 30 of 39)

- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [A Survey on Aspect-Based Sentiment Analysis: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2203.01054/) (2022)
- [BBTv2: Towards a Gradient-Free Future with Large Language Models](https://mlsanity.com/paper/2205.11200/) (2022)
- [MvP: Multi-view Prompting Improves Aspect Sentiment Tuple Prediction](https://mlsanity.com/paper/2305.12627/) (2023)
- [Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training](https://mlsanity.com/paper/2210.15929/) (2022)
- [Towards Efficient NLP: A Standard Evaluation and A Strong Baseline](https://mlsanity.com/paper/2110.07038/) (2021)
- [Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Language](https://mlsanity.com/paper/2303.03363/) (2023)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers](https://mlsanity.com/paper/2211.08025/) (2022)
- [A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation](https://mlsanity.com/paper/2203.01670/) (2022)
- [A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systems](https://mlsanity.com/paper/2204.06923/) (2022)
- [EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training](https://mlsanity.com/paper/2203.09313/) (2022)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts](https://mlsanity.com/paper/2210.11292/) (2022)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification](https://mlsanity.com/paper/2303.07142/) (2023)
- [Generative Knowledge Selection for Knowledge-Grounded Dialogues](https://mlsanity.com/paper/2304.04836/) (2023)
- [Masked Vision-Language Transformer in Fashion](https://mlsanity.com/paper/2210.15110/) (2022)
- [Constrained Sequence-to-Tree Generation for Hierarchical Text Classification](https://mlsanity.com/paper/2204.00811/) (2022)
- [Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning](https://mlsanity.com/paper/2210.07565/) (2022)
- [Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models](https://mlsanity.com/paper/2306.03799/) (2023)
- [Sequence Generation with Label Augmentation for Relation Extraction](https://mlsanity.com/paper/2212.14266/) (2022)
- [Multilingual Normalization of Temporal Expressions with Masked Language Models](https://mlsanity.com/paper/2205.10399/) (2022)
- [Is GPT-3 all you need for Visual Question Answering in Cultural Heritage?](https://mlsanity.com/paper/2207.12101/) (2022)
- [A Kind Introduction to Lexical and Grammatical Aspect, with a Survey of Computational Approaches](https://mlsanity.com/paper/2208.09012/) (2022)
- [Query2Triple: Unified Query Encoding for Answering Diverse Complex Queries over Knowledge Graphs](https://mlsanity.com/paper/2310.11246/) (2023)
- [Knowledge-informed Molecular Learning: A Survey on Paradigm Transfer](https://mlsanity.com/paper/2202.10587/) (2022)

## Abstract (arXiv)

In the era of deep learning, modeling for most NLP tasks has converged to several mainstream paradigms. For example, we usually adopt the sequence labeling paradigm to solve a bundle of tasks such as POS-tagging, NER, Chunking, and adopt the classification paradigm to solve tasks like sentiment analysis. With the rapid progress of pre-trained language models, recent years have observed a rising trend of Paradigm Shift, which is solving one NLP task by reformulating it as another one. Paradigm shift has achieved great success on many tasks, becoming a promising way to improve model performance. Moreover, some of these paradigms have shown great potential to unify a large number of NLP tasks, making it possible to build a single model to handle diverse tasks. In this paper, we review such phenomenon of paradigm shifts in recent years, highlighting several paradigms that have the potential to solve different NLP tasks.