# How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances

arXiv: https://arxiv.org/abs/2310.07343 · HTML: https://mlsanity.com/paper/2310.07343/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 153 of its 196 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 30 of 153)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)

## Built on by (most cited first, 30 of 33)

- [A Comprehensive Study of Knowledge Editing for Large Language Models](https://mlsanity.com/paper/2401.01286/) (2024)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Large Multimodal Agents: A Survey](https://mlsanity.com/paper/2402.15116/) (2024)
- [SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning](https://mlsanity.com/paper/2407.12874/) (2024)
- [Self-Updatable Large Language Models by Integrating Context into Model Parameters](https://mlsanity.com/paper/2410.00487/) (2024)
- [Knowledge Editing in Language Models via Adapted Direct Preference Optimization](https://mlsanity.com/paper/2406.09920/) (2024)
- [GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models](https://mlsanity.com/paper/2312.06315/) (2023)
- [Learning to Poison Large Language Models for Downstream Manipulation](https://mlsanity.com/paper/2402.13459/) (2024)
- [Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers](https://mlsanity.com/paper/2406.18400/) (2024)
- [Towards LifeSpan Cognitive Systems](https://mlsanity.com/paper/2409.13265/) (2024)
- [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025)
- [Model Editing for LLMs4Code: How Far are We?](https://mlsanity.com/paper/2411.06638/) (2024)
- [LLMs as Repositories of Factual Knowledge: Limitations and Solutions](https://mlsanity.com/paper/2501.12774/) (2025)
- [ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains](https://mlsanity.com/paper/2410.09870/) (2024)
- [Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents](https://mlsanity.com/paper/2504.02800/) (2025)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning](https://mlsanity.com/paper/2504.00472/) (2025)
- [Provable Benefits of In-Tool Learning for Large Language Models](https://mlsanity.com/paper/2508.20755/) (2025)
- [STRUCTSURVEY: Structured Agentic Retrieval for Automated Survey Paper Generation](https://mlsanity.com/paper/2607.01243/) (2026)
- [Enhancing Conflict Resolution in Language Models via Abstract Argumentation](https://mlsanity.com/paper/2412.16725/) (2024)
- [KnowledgeSmith: Uncovering Knowledge Updating in LLMs with Model Editing and Unlearning](https://mlsanity.com/paper/2510.02392/) (2025)
- [Adversarial Attacks on Data Attribution](https://mlsanity.com/paper/2409.05657/) (2024)
- [Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction](https://mlsanity.com/paper/2503.06119/) (2025)
- [Mitigating Context-Memory Conflicts in LLMs through Dynamic Cognitive Reconciliation Decoding](https://mlsanity.com/paper/2605.12185/) (2026)
- [Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2504.07754/) (2025)
- [SAGE: Exploring the Boundaries of Unsafe Concept Domain with Semantic-Augment Erasing](https://mlsanity.com/paper/2506.09363/) (2025)
- [Foundational Design Principles and Patterns for Building Robust and Adaptive GenAI-Native Systems](https://mlsanity.com/paper/2508.15411/) (2025)
- [SDA-PLANNER: State-Dependency Aware Adaptive Planner for Embodied Task Planning](https://mlsanity.com/paper/2509.26375/) (2025)
- [Sentiment Analysis in Digital Spaces: An Overview of Reviews](https://mlsanity.com/paper/2310.19687/) (2023)
- [ICA-RAG: Information Completeness Guided Adaptive Retrieval-Augmented Generation for Disease Diagnosis](https://mlsanity.com/paper/2502.14614/) (2025)

## Abstract (arXiv)

Although large language models (LLMs) are impressive in solving various tasks, they can quickly be outdated after deployment. Maintaining their up-to-date status is a pressing concern in the current era. This paper provides a comprehensive review of recent advances in aligning LLMs with the ever-changing world knowledge without re-training from scratch. We categorize research works systemically and provide in-depth comparisons and discussion. We also discuss existing challenges and highlight future directions to facilitate research in this field. We release the paper list at https://github.com/hyintell/awesome-refreshing-llms