# Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models

arXiv: https://arxiv.org/abs/2407.08039 · HTML: https://mlsanity.com/paper/2407.08039/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 74 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 30 of 60)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization](https://mlsanity.com/paper/2311.16839/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Reasoning About Generalization via Conditional Mutual Information](https://mlsanity.com/paper/2001.09122/) (2020)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)

## Built on by (most cited first, 25 of 25)

- [MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation](https://mlsanity.com/paper/2410.11779/) (2024)
- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs](https://mlsanity.com/paper/2502.19148/) (2025)
- [Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations](https://mlsanity.com/paper/2504.12691/) (2025)
- [MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration](https://mlsanity.com/paper/2505.23224/) (2025)
- [From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Education](https://mlsanity.com/paper/2509.21972/) (2025)
- [Integrative Decoding: Improve Factuality via Implicit Self-consistency](https://mlsanity.com/paper/2410.01556/) (2024)
- [ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains](https://mlsanity.com/paper/2410.09870/) (2024)
- [Continual Memorization of Factoids in Language Models](https://mlsanity.com/paper/2411.07175/) (2024)
- [Attributing Culture-Conditioned Generations to Pretraining Corpora](https://mlsanity.com/paper/2412.20760/) (2024)
- [How new data permeates LLM knowledge and how to dilute it](https://mlsanity.com/paper/2504.09522/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [NIRVANA: Structured Pruning Reimagined for Large Language Model Compression](https://mlsanity.com/paper/2509.14230/) (2025)
- [Geometric-disentangelment Unlearning](https://mlsanity.com/paper/2511.17100/) (2025)
- [PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations](https://mlsanity.com/paper/2604.16909/) (2026)
- [Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor](https://mlsanity.com/paper/2507.15903/) (2025)
- [Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B](https://mlsanity.com/paper/2510.08624/) (2025)
- [DePass: Unified Feature Attributing by Simple Decomposed Forward Pass](https://mlsanity.com/paper/2510.18462/) (2025)
- [Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models](https://mlsanity.com/paper/2511.11751/) (2025)
- [LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension](https://mlsanity.com/paper/2511.12020/) (2025)
- [Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures](https://mlsanity.com/paper/2511.13640/) (2025)
- [From Knowledge to Action: Outcomes of the 2025 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry](https://mlsanity.com/paper/2605.03205/) (2026)
- [Exploring Causal Effect of Social Bias on Faithfulness Hallucinations in Large Language Models](https://mlsanity.com/paper/2508.07753/) (2025)
- [HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs](https://mlsanity.com/paper/2605.02443/) (2026)

## Abstract (arXiv)

Hallucination is often regarded as a major impediment for using large language models (LLMs), especially for knowledge-intensive tasks. Even when the training corpus consists solely of true statements, language models still generate hallucinations in the form of amalgamations of multiple facts. We coin this phenomenon as ``knowledge overshadowing'': when we query knowledge from a language model with multiple conditions, some conditions overshadow others, leading to hallucinated outputs. This phenomenon partially stems from training data imbalance, which we verify on both pretrained models and fine-tuned models, over a wide range of LM model families and sizes.From a theoretical point of view, knowledge overshadowing can be interpreted as over-generalization of the dominant conditions (patterns). We show that the hallucination rate grows with both the imbalance ratio (between the popular and unpopular condition) and the length of dominant condition description, consistent with our derived generalization bound. Finally, we propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced, along with a training-free self-contrastive decoding method to alleviate hallucination during inference. Our proposed approach showcases up to 82% F1 for hallucination anticipation and 11.2% to 39.4% hallucination control, with different models and datasets.