# RLP: Reinforcement as a Pretraining Objective

arXiv: https://arxiv.org/abs/2510.01265 · HTML: https://mlsanity.com/paper/2510.01265/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://mlsanity.com/paper/2503.24290/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [Reinforcing General Reasoning without Verifiers](https://mlsanity.com/paper/2505.21493/) (2025)
- [e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs](https://mlsanity.com/paper/2506.09026/) (2025)
- [Reinforcement Pre-Training](https://mlsanity.com/paper/2506.08007/) (2025)
- [Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning](https://mlsanity.com/paper/2504.13941/) (2025)
- [NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning](https://mlsanity.com/paper/2505.16022/) (2025)
- [General Intelligence Requires Reward-based Pretraining](https://mlsanity.com/paper/2502.19402/) (2025)

## Built on by (most cited first, 25 of 25)

- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Entropy After </Think> for reasoning model early exiting](https://mlsanity.com/paper/2509.26522/) (2025)
- [Learning to Simulate Human Dialogue](https://mlsanity.com/paper/2601.04436/) (2026)
- [PretrainZero: Reinforcement Active Pretraining](https://mlsanity.com/paper/2512.03442/) (2025)
- [Self-Improving Pretraining: using post-trained models to pretrain better models](https://mlsanity.com/paper/2601.21343/) (2026)
- [Efficient Process Reward Modeling via Contrastive Mutual Information](https://mlsanity.com/paper/2604.10660/) (2026)
- [Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models](https://mlsanity.com/paper/2603.12248/) (2026)
- [From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space](https://mlsanity.com/paper/2604.14142/) (2026)
- [Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning](https://mlsanity.com/paper/2511.13726/) (2025)
- [Cognitive Training for Language Models: Towards General Capabilities via Cross-Entropy Games](https://mlsanity.com/paper/2603.22479/) (2026)
- [Reinforced Fast Weights with Next-Sequence Prediction](https://mlsanity.com/paper/2602.16704/) (2026)
- [Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning](https://mlsanity.com/paper/2603.09221/) (2026)
- [Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages](https://mlsanity.com/paper/2605.20285/) (2026)
- [RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training](https://mlsanity.com/paper/2606.04272/) (2026)
- [SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation](https://mlsanity.com/paper/2608.09271/) (2026)
- [SemRep: Generative Code Representation Learning with Code Transformations](https://mlsanity.com/paper/2603.13640/) (2026)
- [Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities](https://mlsanity.com/paper/2605.10810/) (2026)
- [Dynamic Latent Routing](https://mlsanity.com/paper/2605.14323/) (2026)
- [Cross-Entropy Games and Frost Training](https://mlsanity.com/paper/2605.27701/) (2026)
- [Multi-Hop Knowledge Composition is Bound by Pretraining Exposure](https://mlsanity.com/paper/2606.09338/) (2026)
- [Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization](https://mlsanity.com/paper/2606.18961/) (2026)
- [Is Next-Chunk Reasoning RL Really Better than SFT? Revisiting Training Strategies under no-CoT Data](https://mlsanity.com/paper/2608.23256/) (2026)
- [T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training](https://mlsanity.com/paper/2609.32791/) (2026)
- [d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Models](https://mlsanity.com/paper/2609.35362/) (2026)
- [Long-Horizon Textual World Modeling through Structured Reasoning](https://mlsanity.com/paper/2610.06637/) (2026)

## Abstract (arXiv)

The dominant paradigm for training large reasoning models starts with pre-training using next-token prediction loss on vast amounts of data. Reinforcement learning, while powerful in scaling reasoning, is introduced only as the very last phase of post-training, preceded by supervised fine-tuning. While dominant, is this an optimal way of training? In this paper, we present RLP, an information-driven reinforcement pretraining objective, that brings the core spirit of reinforcement learning -- exploration -- to the last phase of pretraining. The key idea is to treat chain-of-thought as an exploratory action, with rewards computed based on the information gain it provides for predicting future tokens. This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining. More concretely, the reward signal measures the increase in log-likelihood of the next token when conditioning on both context and a sampled reasoning chain, compared to conditioning on context alone. This approach yields a verifier-free dense reward signal, allowing for efficient training for the full document stream during pretraining. Specifically, RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning. Pretraining with RLP on Qwen3-1.7B-Base lifts the overall average across an eight-benchmark math-and-science suite by 19%. With identical post-training, the gains compound, with the largest improvements on reasoning-heavy tasks such as AIME25 and MMLU-Pro. Applying RLP to the Nemotron-Nano-12B-v2 increases the overall average from 42.81% to 61.32% and raises the average on scientific reasoning by 23%, demonstrating scalability across architectures and model sizes.