mlsanityData to 7 Oct 2026

Tailored Primitive Initialization is the Secret Key to Reinforcement Learning

arXiv 2511.12429 ↗First posted 16 Nov 2025cs.LGcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs). While RL has demonstrated substantial performance gains, it still faces key challenges, including low sampling efficiency and a strong dependence on model initialization: some models achieve rapid improvements with minimal RL steps, while others require significant training data to make progress. In this work, we investigate these challenges through the lens of reasoning token coverage and argue that initializing LLMs with diverse, high-quality reasoning primitives is essential for achieving stable and sample-efficient RL training. We propose Tailor, a finetuning pipeline that automatically discovers and curates novel reasoning primitives, thereby expanding the coverage of reasoning-state distributions before RL. Extensive experiments on mathematical and logical reasoning benchmarks demonstrate that Tailor generates more diverse and higher-quality warm-start data, resulting in higher downstream RL performance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 71 of its 80 references that are in the map. First posted on arXiv 16 Nov 2025.

Builds on

71 of 80 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance9 Aug
  2. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  3. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  6. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  7. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  8. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  9. 2025Behavior Injection: Preparing Language Models for Reinforcement Learning25 May
  10. 2025Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning23 May
  11. 2025Learning to Reason under Off-Policy Guidance21 Apr
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
8 more
  1. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  2. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  6. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  7. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search4 Feb
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
38 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  4. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  5. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  6. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  7. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  8. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  9. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  10. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  11. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  12. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  13. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  14. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  15. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  16. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  17. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  18. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  19. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  20. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  21. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  22. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  23. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  24. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  25. 2024Driving Everywhere with Large Language Model Policy Adaptationcited by 53
  26. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  27. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 35
  28. 2019Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitivescited by 30
  29. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  30. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  31. 2025NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Taskscited by 29
  32. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 29
  33. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  34. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  35. 2025Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineeringcited by 16
  36. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  37. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  38. 2025ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Contextcited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
  2. 2026Learning Generalizable Behaviors for Terminal Agentscited by 1
  3. 2026OraclePhys: A Systematic Framework for LLM Fine-Tuning on Structural Mechanicscited by 0
  4. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  2. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  3. 2025Token-Efficient RL for LLM Reasoning
  4. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
  5. 2025Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
  6. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  8. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection