mlsanityData to 14 Oct 2026

Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings

arXiv 2505.13718 ↗First posted 19 May 2025cs.AI cs.CLcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Designing effective reasoning-capable LLMs typically requires training using Reinforcement Learning with Verifiable Rewards (RLVR) or distillation with carefully curated Long Chain of Thoughts (CoT), both of which depend heavily on extensive training data. This creates a major challenge when the amount of quality training data is scarce. We propose a sample-efficient, two-stage training strategy to develop reasoning LLMs under limited supervision. In the first stage, we "warm up" the model by distilling Long CoTs from a toy domain, namely, Knights & Knaves (K&K) logic puzzles to acquire general reasoning skills. In the second stage, we apply RLVR to the warmed-up model using a limited set of target-domain examples. Our experiments demonstrate that this two-phase approach offers several benefits: (i) the warmup phase alone facilitates generalized reasoning, leading to performance improvements across a range of tasks, including MATH, HumanEval+, and MMLU-Pro; (ii) When both the base model and the warmed-up model are RLVR trained on the same small dataset (≤100 examples), the warmed-up model consistently outperforms the base model; (iii) Warming up before RLVR training allows a model to maintain cross-domain generalizability even after training on a specific domain; (iv) Introducing warmup in the pipeline improves not only accuracy but also overall sample efficiency during RLVR training. The results in this paper highlight the promise of warmup for building robust reasoning LLMs in data-scarce environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 46 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

29 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  7. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  8. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 19

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2025s1: Simple test-time scalingcited by 1,341
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  9. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  10. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  11. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  12. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
7 more
  1. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  2. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  3. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 92
  4. 2025Concise Reasoning via Reinforcement Learningcited by 79
  5. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  6. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  7. 2024Meta Reasoning for Large Language Modelscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignmentcited by 13
  2. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  3. 2026One Sample to Rule Them All: Extreme Data Efficiency in Multidiscipline Reasoning with Reinforcement Learningcited by 2
  4. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  4. 2026Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
  5. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
  6. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning
  7. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning
  8. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection