mlsanityData to 7 Oct 2026

The Life Cycle of Knowledge in Big Language Models: A Survey

arXiv 2303.07616 ↗First posted 14 Mar 2023cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Knowledge plays a critical role in artificial intelligence. Recently, the extensive success of pre-trained language models (PLMs) has raised significant attention about how knowledge can be acquired, maintained, updated and used by language models. Despite the enormous amount of related studies, there still lacks a unified view of how knowledge circulates within language models throughout the learning, tuning, and application processes, which may prevent us from further understanding the connections between current progress or realizing existing limitations. In this survey, we revisit PLMs as knowledge-based systems by dividing the life circle of knowledge in PLMs into five critical periods, and investigating how knowledge circulates when it is built, maintained and used. To this end, we systematically review existing studies of each period of the knowledge life cycle, summarize the main challenges and current limitations, and discuss future directions.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 144 of its 177 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

144 of 177 references.

References · 144

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  13. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  14. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  15. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  16. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  17. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  18. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  19. 2016HyperNetworkscited by 1,003
  20. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  21. 2019How Can We Know What Language Models Know?cited by 953
  22. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  23. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  24. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  25. 2017Counterfactual Fairnesscited by 802
  26. 2021GPT Understands, Toocited by 715
  27. 2022Transformers learn in-context by gradient descentcited by 714
  28. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  29. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  30. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  31. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  32. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  33. 2021Editing Factual Knowledge in Language Modelscited by 548
  34. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  35. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  36. 2021Fast Model Editing at Scalecited by 491
  37. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  38. 2022Memory-Based Model Editing at Scalecited by 443

50 most cited of 144 All 144 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 12; 2025: 3; 2026: 1.

23 papers.

Sort:

By area:

  1. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  2. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  3. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  4. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  5. 2023Large Language Models are Zero Shot Hypothesis Proposerscited by 45
  6. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  7. 2024Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluationcited by 33
  8. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  9. 2023DocTER: Evaluating Document-based Knowledge Editingcited by 26
  10. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 22
  11. 2023Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Modelscited by 18
  12. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
11 more
  1. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
  2. 2024Knowledge Localization: Mission Not Accomplished? Enter Query Localization!cited by 13
  3. 2024BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Modelscited by 11
  4. 2023Enhancing LLM with Evolutionary Fine Tuning for News Summary Generationcited by 10
  5. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  6. 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
  7. 2024Zero-Shot Cross-Lingual Document-Level Event Causality Identification with Heterogeneous Graph Contrastive Transfer Learningcited by 3
  8. 2024Profiling Bias in LLMs: Stereotype Dimensions in Contextual Word Embeddingscited by 3
  9. 2024One Mind, Many Tongues: A Deep Dive into Language-Agnostic Knowledge Neurons in Large Language Modelscited by 3
  10. 2025Reformulation is All You Need: Addressing Malicious Text Features in DNNscited by 1
  11. 2026Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Modelscited by 1