mlsanityData to 7 Oct 2026

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

arXiv 2111.01243 ↗First posted 1 Nov 2021cs.CL cs.AI cs.LGcited by 457

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.

Key facts. Cited by 457 papers in the map (data through 7 Oct 2026). Builds on 261 of its 363 references that are in the map. First posted on arXiv 1 Nov 2021.

Builds on

261 of 363 references.

References · 261

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  13. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  14. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  15. 2019Language Models as Knowledge Bases?cited by 1,922
  16. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  17. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  18. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  19. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  20. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  21. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  22. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  23. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  24. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  25. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  26. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  27. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  28. 2014How transferable are features in deep neural networks?cited by 1,216
  29. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  30. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  31. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  32. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  33. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  34. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  35. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  36. 2019Publicly Available Clinical BERT Embeddingscited by 995
  37. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  38. 2019How Can We Know What Language Models Know?cited by 953

50 most cited of 261 All 261 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 112; 2024: 208; 2025: 93; 2026: 23.

457 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  6. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  7. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  9. 2023A Cookbook of Self-Supervised Learningcited by 248
  10. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  11. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  12. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
38 more
  1. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  2. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  3. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2023All in One: Multi-task Prompting for Graph Neural Networkscited by 147
  7. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  8. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  9. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  10. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  11. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  12. 2023Large Language Models in Law: A Surveycited by 128
  13. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  14. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 116
  15. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  16. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  17. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  18. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  19. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  20. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  21. 2024NetLLM: Adapting Large Language Models for Networkingcited by 76
  22. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  23. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  24. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  25. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  26. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  27. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  28. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  29. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  30. 2023Large Generative AI Models for Telecom: The Next Big Thing?cited by 52
  31. 2023Algorithm Evolution Using Large Language Modelcited by 52
  32. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  33. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  34. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  35. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  36. 2024LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendationcited by 47
  37. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  38. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?cited by 44

50 most cited of 457 All 457 (JSON)