mlsanityData to 14 Oct 2026

Behavior Injection: Preparing Language Models for Reinforcement Learning

arXiv 2505.18917 ↗First posted 25 May 2025cs.LG cs.AIcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has emerged as a powerful post-training technique to incentivize the reasoning ability of large language models (LLMs). However, LLMs can respond very inconsistently to RL finetuning: some show substantial performance gains, while others plateau or even degrade. To understand this divergence, we analyze the per-step influence of the RL objective and identify two key conditions for effective post-training: (1) RL-informative rollout accuracy, and (2) strong data co-influence, which quantifies how much the training data affects performance on other samples. Guided by these insights, we propose behavior injection, a task-agnostic data augmentation scheme applied prior to RL. Behavior injection enriches the supervised finetuning (SFT) data by seeding exploratory and exploitative behaviors, effectively making the model more RL-ready. We evaluate our method across two reasoning benchmarks with multiple base models. The results demonstrate that our theoretically motivated augmentation can significantly increase the performance gain from RL over the pre-RL model.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 92 of its 105 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

92 of 105 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  3. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  8. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  9. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  10. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  11. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search4 Feb
  12. 2025Process Reinforcement through Implicit Rewards3 Feb
3 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 77

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2024Qwen2.5 Technical Reportcited by 4,942
38 more
  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  12. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  13. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  14. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  15. 2016Deep Exploration via Bootstrapped DQNcited by 527
  16. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  17. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  18. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  19. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  20. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
  21. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  22. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  23. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  24. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  25. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  26. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  27. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  28. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  29. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  30. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  31. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
  32. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  33. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  34. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
  35. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  36. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  37. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
  38. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 28
  2. 2025Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levelscited by 8
  3. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  4. 2026RL Starts before RL: On Policy Distillation for Better Reinforcement Learningcited by 3
  5. 2026ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoningcited by 2
  6. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  7. 2026Learning Generalizable Behaviors for Terminal Agentscited by 2
  8. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  2. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  3. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  4. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
  5. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  6. 2025Reinforcement Pre-Training
  7. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models
  8. 2026Demystifying Reinforcement Learning Post-Training of Language Models