Layer of Truth: Probing Belief Shifts under Continual Pre-Training Poisoning
Not on the map. Why?
Abstract (arXiv)
We show that continual pretraining on plausible misinformation can overwrite specific factual knowledge in large language models without degrading overall performance. Unlike prior poisoning work under static pretraining, we study repeated exposure to counterfactual claims during continual updates. Using paired fact-counterfact items with graded poisoning ratios, we track how internal preferences between competing facts evolve across checkpoints, layers, and model scales. Even moderate poisoning (50-100%) flips over 55% of responses from correct to counterfactual while leaving ambiguity nearly unchanged. These belief flips emerge abruptly, concentrate in late layers (e.g., Layers 29-36 in 3B models), and are partially reversible via patching (up to 56.8%). The corrupted beliefs generalize beyond poisoned prompts, selectively degrading commonsense reasoning while leaving alignment benchmarks largely intact and transferring imperfectly across languages. These results expose a failure mode of continual pre-training in which targeted misinformation replaces internal factual representations without triggering broad performance collapse, motivating representation-level monitoring of factual integrity during model updates.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 29 Oct 2025.
Builds on
20 of 36 references.
References · 20
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
- 2023The Internal State of an LLM Knows When It's Lyingcited by 659
- 2021Knowledge Neurons in Pretrained Transformerscited by 601
8 more
- 2019Defending Against Neural Fake Newscited by 559
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
- 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
- 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
- 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 72
- 2023Characterizing Mechanisms for Factual Recall in Language Modelscited by 48
- 2024Persistent Pre-Training Poisoning of LLMscited by 48
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
- 2026Reasoning or Rationalization? The Role of Justifications in Masked Diffusion Models for Fact Verification
- 2026The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models
- 2026STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
- 2025Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models
- 2026PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning
- 2026Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning
- 2026Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetry