mlsanityData to 14 Oct 2026

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining

arXiv 2504.07912 ↗First posted 10 Apr 2025cs.LGcited by 129

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models for advanced mathematical reasoning and coding. Following the success of frontier reasoning models, recent work has demonstrated that RL fine-tuning consistently improves performance, even in smaller-scale models; however, the underlying mechanisms driving these improvements are not well-understood. Understanding the effects of RL fine-tuning requires disentangling its interaction with pretraining data composition, hyperparameters, and model scale, but such problems are exacerbated by the lack of transparency regarding the training data used in many existing models. In this work, we present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch on different mixtures of fully open datasets. We investigate the effects of various RL fine-tuning algorithms (PPO, GRPO, and Expert Iteration) across models of different scales. Our study reveals that RL algorithms consistently converge towards a dominant output distribution, amplifying patterns in the pretraining data. We also find that models of different scales trained on the same data mixture will converge to distinct output distributions, suggesting that there are scale-dependent biases in model generalization. Moreover, we find that RL post-training on simpler questions can lead to performance gains on harder ones, indicating that certain reasoning capabilities generalize across tasks. Our findings show that small-scale proxies in controlled settings can elicit interesting insights regarding the role of RL in shaping language model behavior.

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 64 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

50 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2024Free Process Rewards without Process Labels2 Dec
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  9. 2023Let's Verify Step by Step31 May
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  12. 2022Solving math word problems with process- and outcome-based feedback25 Nov
2 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul
  2. 2017Thinking Fast and Slow with Deep Learning and Tree Search23 May

Other references · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
24 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2020GLU Variants Improve Transformercited by 1,804
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  6. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2024Phi-4 Technical Reportcited by 711
  10. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  13. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  14. 20242 OLMo 2 Furiouscited by 340
  15. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  16. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  17. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  18. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  19. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  20. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  21. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  22. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  23. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  24. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 65; 2026: 64.

129 papers.

Sort:

By area:

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  2. 2025Learning to Reason under Off-Policy Guidancecited by 281
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 97
  6. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  7. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  8. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  9. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  10. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  11. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  12. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
38 more
  1. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Explorationcited by 43
  2. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 42
  3. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  4. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  5. 2026Maximum Likelihood Reinforcement Learningcited by 36
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  7. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 35
  8. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  9. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  10. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  11. 2025Base Models Know How to Reason, Thinking Models Learn Whencited by 27
  12. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  13. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 24
  14. 2025Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoningcited by 23
  15. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
  16. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  17. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  18. 2025CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generationcited by 19
  19. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Modelscited by 18
  20. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  21. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 18
  22. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  23. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Themcited by 15
  24. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
  25. 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoningcited by 14
  26. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 14
  27. 2025MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancementcited by 14
  28. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 14
  29. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoningcited by 13
  30. 2025ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learningcited by 13
  31. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  32. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  33. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 12
  34. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  35. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  36. 2026ERNIE 5.0 Technical Reportcited by 11
  37. 2026Entropy-Preserving Reinforcement Learningcited by 11
  38. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement learning fine-tuning of language model for instruction following and math reasoning
  2. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
  3. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
  4. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
  5. 2026Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
  6. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  7. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  8. 2026Understanding Reasoning from Pretraining to Post-Training