# On Language Drift during RLVR Post-Training

arXiv: https://arxiv.org/abs/2610.02015 · HTML: https://mlsanity.com/paper/2610.02015/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning](https://mlsanity.com/paper/2005.07064/) (2020)
- [DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks](https://mlsanity.com/paper/2607.07946/) (2026)
- [Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought](https://mlsanity.com/paper/2603.05488/) (2026)
- [Reasoning Models Sometimes Output Illegible Chains of Thought](https://mlsanity.com/paper/2510.27338/) (2025)
- [Stealing Reasoning Traces from Proprietary LLM APIs](https://mlsanity.com/paper/2608.09867/) (2026)
- [Success and Cost Elicit Convention Formation for Efficient Communication](https://mlsanity.com/paper/2510.24023/) (2025)
- [Measuring Weak-to-Strong Legibility of Reasoning Models](https://mlsanity.com/paper/2603.20508/) (2026)
- [Not All LLM Reasoning is Visible in the Chain-of-Thought](https://mlsanity.com/paper/2607.22925/) (2026)
- [Large Language Models Do Not Always Need Readable Language](https://mlsanity.com/paper/2606.19857/) (2026)

## Abstract (arXiv)

Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.