mlsanityData to 14 Oct 2026

Calibrating Factual Knowledge in Pretrained Language Models

arXiv 2210.03329 ↗First posted 7 Oct 2022cs.CLcited by 92

Not on the map. Why?

Abstract (arXiv)

Previous literature has proved that Pretrained Language Models (PLMs) can store factual knowledge. However, we find that facts stored in the PLMs are not always correct. It motivates us to explore a fundamental question: How do we calibrate factual knowledge in PLMs without re-training from scratch? In this work, we propose a simple and lightweight method CaliNet to achieve this goal. To be specific, we first detect whether PLMs can learn the right facts via a contrastive score between right and fake facts. If not, we then use a lightweight method to add and adapt new parameters to specific factual texts. Experiments on the knowledge probing task show the calibration effectiveness and efficiency. In addition, through closed-book question answering, we find that the calibrated PLM possesses knowledge generalization ability after fine-tuning. Beyond the calibration performance, we further investigate and visualize the knowledge calibration mechanism.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

22 of 26 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2019Language Models as Knowledge Bases?cited by 1,936
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2019How Can We Know What Language Models Know?cited by 959
  9. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  11. 2021Editing Factual Knowledge in Language Modelscited by 553
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
10 more
  1. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  2. 2021Fast Model Editing at Scalecited by 498
  3. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  4. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  5. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  6. 2020Modifying Memories in Transformer Modelscited by 224
  7. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  8. 2019Inducing Relational Knowledge from BERTcited by 75
  9. 2020Language Models as Fact Checkers?cited by 51
  10. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 27; 2024: 40; 2025: 17; 2026: 8.

92 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  3. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  4. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 285
  5. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  6. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  7. 2023Knowledge Editing for Large Language Models: A Surveycited by 194
  8. 2025MemOS: A Memory OS for AI Systemcited by 138
  9. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  10. 2023Do Large Language Models Know about Facts?cited by 51
  11. 2023Learning from models beyond fine-tuningcited by 48
  12. 2023Can We Edit Multimodal Large Language Models?cited by 44
38 more
  1. 2024WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editingcited by 40
  2. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  3. 2024IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neuronscited by 38
  4. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 36
  5. 2023Evaluating the Reversal Curse in Model Editingcited by 34
  6. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  7. 2024In-Context Editing: Learning Knowledge from Self-Induced Distributionscited by 34
  8. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  9. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  10. 2024Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearningcited by 29
  11. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  12. 2025The Mirage of Model Editing: Revisiting Evaluation in the Wildcited by 27
  13. 2023DocTER: Evaluating Document-based Knowledge Editingcited by 26
  14. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 25
  15. 2023Editing Language Model-based Knowledge Graph Embeddingscited by 23
  16. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  17. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  18. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 21
  19. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  20. 2024Neuron-Level Sequential Editing for Large Language Modelscited by 20
  21. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19
  22. 2025Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuningcited by 19
  23. 2024LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environmentscited by 18
  24. 2024InstructEdit: Instruction-based Knowledge Editing for Large Language Modelscited by 17
  25. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  26. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  27. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  28. 2024Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Modelscited by 14
  29. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  30. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimizationcited by 14
  31. 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awarenesscited by 14
  32. 2023Assessing the Reliability of Large Language Model Knowledgecited by 13
  33. 2023Retrieval-augmented Multilingual Knowledge Editingcited by 13
  34. 2024Stable Knowledge Editing in Large Language Modelscited by 13
  35. 2024Towards LifeSpan Cognitive Systemscited by 13
  36. 2024Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answeringcited by 13
  37. 2024Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Modelscited by 12
  38. 2024LeKUBE: A Legal Knowledge Update BEnchmarkcited by 12

50 most cited of 92 All 92 (JSON)