# Information Association for Language Model Updating by Mitigating LM-Logical Discrepancy

arXiv: https://arxiv.org/abs/2305.18582 · HTML: https://mlsanity.com/paper/2305.18582/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Memory-Based Model Editing at Scale](https://mlsanity.com/paper/2206.06520/) (2022)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs](https://mlsanity.com/paper/2111.13654/) (2021)

## Built on by (most cited first, 10 of 10)

- [SMART: Self-Aware Agent for Tool Overuse Mitigation](https://mlsanity.com/paper/2502.11435/) (2025)
- [How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances](https://mlsanity.com/paper/2310.07343/) (2023)
- ["Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphs](https://mlsanity.com/paper/2309.08594/) (2023)
- [Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansion](https://mlsanity.com/paper/2502.12598/) (2025)
- [From Evidence to Belief: A Bayesian Epistemology Approach to Language Models](https://mlsanity.com/paper/2504.19622/) (2025)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [Is Extending Modality The Right Path Towards Omni-Modality?](https://mlsanity.com/paper/2506.01872/) (2025)
- [Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education](https://mlsanity.com/paper/2309.13059/) (2023)
- [Establishing Knowledge Preference in Language Models](https://mlsanity.com/paper/2407.13048/) (2024)
- [Pretraining Exposure Explains Popularity Judgments in Large Language Models](https://mlsanity.com/paper/2605.12382/) (2026)

## Abstract (arXiv)

Large Language Models~(LLMs) struggle with providing current information due to the outdated pre-training data. Existing methods for updating LLMs, such as knowledge editing and continual fine-tuning, have significant drawbacks in generalizability of new information and the requirements on structured updating corpus. We identify the core challenge behind these drawbacks: the LM-logical discrepancy featuring the difference between language modeling probabilities and logical probabilities. To evaluate and address the core challenge, we propose a new task formulation of the information updating task that only requires the provision of an unstructured updating corpus and evaluates the performance of information updating on the generalizability to question-answer pairs pertaining to the updating information. We further propose a novel and effective pipeline approach for the task, highlighting a self-prompting-based question-answer generation process and a associative distillation methods to bridge the LM-logical discrepancy. We develop two datasets for evaluation, one sourced from news articles published in March and April 2023, and the other from the Natural Questions benchmark. Experimental results demonstrate the superiority of our approach, significantly increasing the factual consistency score (on a scale from 0 to 1) by up to 0.16. Furthermore, our method effectively mitigates forgetting utilizing a compact replay buffer with only 2.3% of the training tokens.