mlsanityData to 7 Oct 2026

Understanding Finetuning for Factual Knowledge Extraction from Language Models

arXiv 2301.11293 ↗First posted 26 Jan 2023cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) pretrained on large corpora of text from the web have been observed to contain large amounts of various types of knowledge about the world. This observation has led to a new and exciting paradigm in knowledge graph construction where, instead of manual curation or text mining, one extracts knowledge from the parameters of an LM. Recently, it has been shown that finetuning LMs on a set of factual knowledge makes them produce better answers to queries from a different set, thus making finetuned LMs a good candidate for knowledge extraction and, consequently, knowledge graph construction. In this paper, we analyze finetuned LMs for factual knowledge extraction. We show that along with its previously known positive effects, finetuning also leads to a (potentially harmful) phenomenon which we call Frequency Shock, where at the test time the model over-predicts rare entities that appear in the training set and under-predicts common entities that do not appear in the training set enough times. We show that Frequency Shock leads to a degradation in the predictions of the model and beyond a point, the harm from Frequency Shock can even outweigh the positive effects of finetuning, making finetuning harmful overall. We then consider two solutions to remedy the identified negative effect: 1- model mixing and 2- mixture finetuning with the LM's pre-training task. The two solutions combined lead to significant improvements compared to vanilla finetuning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

35 of 49 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  5. 2019How Can We Know What Language Models Know?cited by 953
  6. 2016Complex Embeddings for Simple Link Predictioncited by 897
  7. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  8. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  9. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  10. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  11. 2018SimplE Embedding for Link Prediction in Knowledge Graphscited by 195
  12. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
23 more
  1. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  2. 2022A Review on Language Models as Knowledge Basescited by 156
  3. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  4. 2020Tensor Decompositions for temporal knowledge base completioncited by 115
  5. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  6. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  7. 2019Diachronic Embedding for Temporal Knowledge Graph Completioncited by 104
  8. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  9. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  10. 2020Roses Are Red, Violets Are Blue... but Should Vqa Expect Them To?cited by 69
  11. 2020MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answeringcited by 63
  12. 2020Do Language Embeddings Capture Scales?cited by 63
  13. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  14. 2020GenericsKB: A Knowledge Base of Generic Statementscited by 51
  15. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  16. 2021Analyzing Commonsense Emergence in Few-shot Knowledge Modelscited by 34
  17. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  18. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  19. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  20. 2020Differentiable Open-Ended Commonsense Reasoningcited by 29
  21. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
  22. 2022Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptionscited by 17
  23. 2022Short-answer scoring with ensembles of pretrained language modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 3; 2026: 1.

12 papers.

  1. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  2. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  3. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  4. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  5. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 31
  6. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  7. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  8. 2024An Efficient Rehearsal Scheme for Catastrophic Forgetting Mitigation during Multi-stage Fine-tuningcited by 4
  9. 2025Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridgecited by 3
  10. 2025ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrievalcited by 1
  11. 2026Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Modelscited by 1
  12. 2025From Parameters to Prompts: Understanding and Mitigating the Factuality Gap between Fine-Tuned LLMscited by 0