mlsanityData to 7 Oct 2026

Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models

arXiv 2410.09629 ↗First posted 12 Oct 2024cs.CL cs.AI cs.IRcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are proficient in capturing factual knowledge across various domains. However, refining their capabilities on previously seen knowledge or integrating new knowledge from external sources remains a significant challenge. In this work, we propose a novel synthetic knowledge ingestion method called Ski, which leverages fine-grained synthesis, interleaved generation, and assemble augmentation strategies to construct high-quality data representations from raw knowledge sources. We then integrate Ski and its variations with three knowledge injection techniques: Retrieval Augmented Generation (RAG), Supervised Fine-tuning (SFT), and Continual Pre-training (CPT) to inject and refine knowledge in language models. Extensive empirical experiments are conducted on various question-answering tasks spanning finance, biomedicine, and open-generation domains to demonstrate that Ski significantly outperforms baseline methods by facilitating effective knowledge injection. We believe that our work is an important step towards enhancing the factual accuracy of LLM outputs by refining knowledge representation and injection capabilities.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

32 of 41 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
20 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,080
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  5. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  6. 2023Fine-tuning Language Models for Factualitycited by 219
  7. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  8. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  9. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  10. 2024Continual Learning for Large Language Models: A Surveycited by 144
  11. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  12. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  13. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  14. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  15. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 80
  16. 2024Injecting New Knowledge into Large Language Models via Supervised Fine-Tuningcited by 56
  17. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  18. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  19. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systemscited by 24
  20. 2023KILM: Knowledge Injection into Encoder-Decoder Language Modelscited by 19

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  2. 2025A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Application: Data Utility and Quality Perspectivescited by 3
  3. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regimecited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG
  2. 2025GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
  3. 2025Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
  4. 2025Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions
  5. 2026Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Models
  6. 2026PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMs
  7. 2024Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answering
  8. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains