mlsanityData to 7 Oct 2026

KBLaM: Knowledge Base augmented Language Model

arXiv 2410.10450 ↗First posted 14 Oct 2024cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose Knowledge Base augmented Language Model (KBLaM), a new method for augmenting Large Language Models (LLMs) with external knowledge. KBLaM works with a knowledge base (KB) constructed from a corpus of documents, transforming each piece of knowledge in the KB into continuous key-value vector pairs via pre-trained sentence encoders with linear adapters and integrating them into pre-trained LLMs via a specialized rectangular attention mechanism. Unlike Retrieval-Augmented Generation, KBLaM eliminates external retrieval modules, and unlike in-context learning, its computational overhead scales linearly with KB size rather than quadratically. Our approach enables integrating a large KB of more than 10K triples into an 8B pre-trained LLM of only 8K context window on one single A100 80GB GPU and allows for dynamic updates without model fine-tuning or retraining. Experiments demonstrate KBLaM's effectiveness in various tasks, including question-answering and open-ended reasoning, while providing interpretable insights into its use of the augmented knowledge. Code and datasets are available at https://github.com/microsoft/KBLaM/

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
25 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  4. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  6. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2022Efficiently Scaling Transformer Inferencecited by 529
  10. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  11. 2022Memorizing Transformerscited by 219
  12. 2022Learning Video Representations from Large Language Modelscited by 204
  13. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  14. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  15. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  16. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  17. 2022Parallel Context Windows for Large Language Modelscited by 81
  18. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  19. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 52
  20. 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
  21. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  22. 2022Memorization in NLP Fine-tuning Methodscited by 43
  23. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  24. 2024Vector-ICL: In-context Learning with Continuous Vector Representationscited by 13
  25. 2023Scaling In-Context Demonstrations with Structured Attentioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  2. 2025LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Taskscited by 7
  3. 2026KV-Skill: Forging Expertise in the Model's Native Languagecited by 3
  4. 2025SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attentioncited by 2
  5. 2026Architecture Matters: Comparing RAG Systems under Knowledge Base Poisoningcited by 2
  6. 2025Knowledge Fusion via Bidirectional Information Aggregationcited by 1
  7. 2025Understanding Parametric Knowledge Injection in Retrieval-Augmented Generationcited by 1
  8. 2025AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAMcited by 1
  9. 2025Automating Violence Detection and Categorization from Ancient Textscited by 0
  10. 2025Accelerating Language Model Workflows with Prompt Choreographycited by 0
  11. 2026Scalable Model-Based Clustering with Sequential Monte Carlocited by 0
  12. 2026KARLA: Knowledge-base Augmented Retrieval for Language Modelscited by 0
4 more
  1. 2026KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Modelscited by 0
  2. 2026Co-LMLM: Continuous-Query Limited Memory Language Modelscited by 0
  3. 2026TokenMem: Faithful Knowledge Injection for Frozen LLMscited by 0
  4. 2026RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models
  2. 2024A Learn-Then-Reason Model Towards Generalization in Knowledge Base Question Answering
  3. 2024Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization
  4. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
  5. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  6. 2024Retrieval-Augmented Language Model for Extreme Multi-Label Knowledge Graph Link Prediction
  7. 2025Knowledge Graph-extended Retrieval Augmented Generation for Question Answering
  8. 2026Knowledge Capsules: Structured Nonparametric Memory Units for LLMs