mlsanityData to 7 Oct 2026

Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications

arXiv 2311.05876 ↗First posted 10 Nov 2023cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit superior performance on various natural language tasks, but they are susceptible to issues stemming from outdated data and domain-specific limitations. In order to address these challenges, researchers have pursued two primary strategies, knowledge editing and retrieval augmentation, to enhance LLMs by incorporating external information from different aspects. Nevertheless, there is still a notable absence of a comprehensive survey. In this paper, we propose a review to discuss the trends in integration of knowledge and large language models, including taxonomy of methods, benchmarks, and applications. In addition, we conduct an in-depth analysis of different methods and point out potential research directions in the future. We hope this survey offers the community quick access and a comprehensive overview of this research area, with the intention of inspiring future research endeavors.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 90 of its 125 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

90 of 125 references.

References · 90

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
38 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  6. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2022Mass-Editing Memory in a Transformercited by 976
  10. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  13. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  14. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  15. 2021Editing Factual Knowledge in Language Modelscited by 548
  16. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  17. 2022Memory-Based Model Editing at Scalecited by 443
  18. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  19. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  20. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  21. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  22. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  23. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  24. 2020Autoregressive Entity Retrievalcited by 333
  25. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  26. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  27. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  28. 2023Are aligned neural networks adversarially aligned?cited by 305
  29. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  30. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  31. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  32. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  33. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  34. 2022Prompting GPT-3 To Be Reliablecited by 258
  35. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  36. 2023Do Large Language Models Know What They Don't Know?cited by 238
  37. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  38. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 21; 2025: 4.

28 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  6. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agentscited by 61
  7. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  8. 2025Do as We Do, Not as You Think: the Conformity of Large Language Modelscited by 48
  9. 2024Dataflow-Guided Retrieval Augmentation for Repository-Level Code Completioncited by 40
  10. 2023Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate Systemcited by 26
  11. 2024Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Checkcited by 24
  12. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
16 more
  1. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  2. 2024Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysiscited by 14
  3. 2024Learning Fine-Grained Grounded Citations for Attributed Large Language Modelscited by 12
  4. 2024Editing Factual Knowledge and Explanatory Ability of Medical Large Language Modelscited by 10
  5. 2024Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answeringcited by 10
  6. 2024Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Modelscited by 9
  7. 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insightscited by 7
  8. 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
  9. 2024Knowledge Editing on Black-box Large Language Modelscited by 4
  10. 2024Precision Knowledge Editing: Enhancing Safety in Large Language Modelscited by 2
  11. 2024Augmenting Textual Generation via Topology Aware Retrievalcited by 1
  12. 2024The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Modelscited by 1
  13. 2025TrustRAG: An Information Assistant with Retrieval Augmented Generationcited by 1
  14. 2025REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editingcited by 1
  15. 2024KaLM: Knowledge-aligned Autoregressive Language Modeling via Dual-view Knowledge Graph Contrastive Learningcited by 0
  16. 2025One for All: Update Parameterized Knowledge Across Multiple Modelscited by 0