# PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning

arXiv: https://arxiv.org/abs/2602.01875 · HTML: https://mlsanity.com/paper/2602.01875/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 28 of 28)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [The Hallucination Tax of Reinforcement Finetuning](https://mlsanity.com/paper/2505.13988/) (2025)
- [Reinforcement Pre-Training](https://mlsanity.com/paper/2506.08007/) (2025)
- [Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism](https://mlsanity.com/paper/2311.01041/) (2023)
- [Shadow-FT: Tuning Instruct Model via Training on Paired Base Model](https://mlsanity.com/paper/2505.12716/) (2025)
- [ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph](https://mlsanity.com/paper/2503.15990/) (2025)

## Built on by (most cited first, 1 of 1)

- [Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors](https://mlsanity.com/paper/2607.00447/) (2026)

## Abstract (arXiv)

Large language models (LLMs), despite their powerful capabilities, suffer from factual hallucinations where they generate verifiable falsehoods. We identify a root of this issue: the imbalanced data distribution in the pretraining corpus, which leads to a state of "low-probability truth" and "high-probability falsehood". Recent approaches, such as teaching models to say "I don't know" or post-hoc knowledge editing, either evade the problem or face catastrophic forgetting. To address this issue from its root, we propose PretrainRL, a novel framework that integrates reinforcement learning into the pretraining phase to consolidate factual knowledge. The core principle of PretrainRL is "debiasing then learning." It actively reshapes the model's probability distribution by down-weighting high-probability falsehoods, thereby making "room" for low-probability truths to be learned effectively. To enable this, we design an efficient negative sampling strategy to discover these high-probability falsehoods and introduce novel metrics to evaluate the model's probabilistic state concerning factual knowledge. Extensive experiments on three public benchmarks demonstrate that PretrainRL significantly alleviates factual hallucinations and outperforms state-of-the-art methods.