# Reinforcement Pre-Training

arXiv: https://arxiv.org/abs/2506.08007 · HTML: https://mlsanity.com/paper/2506.08007/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://mlsanity.com/paper/2504.20571/) (2025)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [General-Reasoner: Advancing LLM Reasoning Across All Domains](https://mlsanity.com/paper/2505.14652/) (2025)
- [Reinforcing General Reasoning without Verifiers](https://mlsanity.com/paper/2505.21493/) (2025)
- [Think Only When You Need with Large Hybrid-Reasoning Models](https://mlsanity.com/paper/2505.14631/) (2025)
- [On-Policy RL with Optimal Reward Baseline](https://mlsanity.com/paper/2505.23585/) (2025)

## Built on by (most cited first, 30 of 40)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey on Latent Reasoning](https://mlsanity.com/paper/2507.06203/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [Generalist Reward Models: Found Inside Large Language Models](https://mlsanity.com/paper/2506.23235/) (2025)
- [RLP: Reinforcement as a Pretraining Objective](https://mlsanity.com/paper/2510.01265/) (2025)
- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning](https://mlsanity.com/paper/2507.17842/) (2025)
- [Reinforcement Learning on Pre-Training Data](https://mlsanity.com/paper/2509.19249/) (2025)
- [Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models](https://mlsanity.com/paper/2510.01304/) (2025)
- [Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels](https://mlsanity.com/paper/2510.06499/) (2025)
- [PretrainZero: Reinforcement Active Pretraining](https://mlsanity.com/paper/2512.03442/) (2025)
- [Likelihood-Based Reward Designs for General LLM Reasoning](https://mlsanity.com/paper/2602.03979/) (2026)
- [Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting](https://mlsanity.com/paper/2605.19444/) (2026)
- [Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts](https://mlsanity.com/paper/2507.19477/) (2025)
- [Reinforcement Mid-Training](https://mlsanity.com/paper/2509.24375/) (2025)
- [Self-Improving Pretraining: using post-trained models to pretrain better models](https://mlsanity.com/paper/2601.21343/) (2026)
- [Thinking Augmented Pre-training](https://mlsanity.com/paper/2509.20186/) (2025)
- [Learning to Reason as Action Abstractions with Scalable Mid-Training RL](https://mlsanity.com/paper/2509.25810/) (2025)
- [DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model](https://mlsanity.com/paper/2512.12633/) (2025)
- [Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models](https://mlsanity.com/paper/2603.12248/) (2026)
- [From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space](https://mlsanity.com/paper/2604.14142/) (2026)
- [BOW: Training Language Models to Reason Over Plausible Next Words](https://mlsanity.com/paper/2506.13502/) (2025)
- [Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production](https://mlsanity.com/paper/2510.13879/) (2025)
- [Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning](https://mlsanity.com/paper/2511.13726/) (2025)
- [Cognitive Training for Language Models: Towards General Capabilities via Cross-Entropy Games](https://mlsanity.com/paper/2603.22479/) (2026)
- [Diversity or Precision? A Deep Dive into Next Token Prediction](https://mlsanity.com/paper/2512.22955/) (2025)
- [PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning](https://mlsanity.com/paper/2602.01875/) (2026)
- [Reinforced Fast Weights with Next-Sequence Prediction](https://mlsanity.com/paper/2602.16704/) (2026)
- [Towards Better RL Training Data Utilization via Second-Order Rollout](https://mlsanity.com/paper/2602.22765/) (2026)

## Abstract (arXiv)

In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.