mlsanityData to 14 Oct 2026

Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond

arXiv 2503.10460 ↗First posted 13 Mar 2025cs.CL cs.LGcited by 175

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

This paper introduces Light-R1, an open-source suite for training long reasoning models using reproducible and cost-effective methodology. Given the proprietary nature of data used in the DeepSeek-R1 series, we develop an alternative approach leveraging exclusively public data and models. Our curriculum training progressively increases data difficulty, combined with multi-staged post-training. Our Light-R1-32B model, trained from Qwen2.5-32B-Instruct, outperforms DeepSeek-R1-Distill-Qwen-32B in math reasoning. Experimental results show that this curriculum approach becomes more effective when distinct, diverse datasets are available for different training stages: fine-tuning DeepSeek-R1-Distilled models (pre-tuned by DeepSeek team on proprietary data) with 3,000 challenging examples from our curriculum dataset yielded state-of-the-art 7B and 14B models, while the 32B model, Light-R1-32B-DS performed comparably to QwQ-32B and DeepSeek-R1. Furthermore, we extend our work by applying GRPO on long reasoning models. Our final Light-R1-14B-DS achieves SOTA performance among 14B models in math, with AIME24 & 25 scores of 74.0 and 60.2 respectively, surpassing many 32B models and DeepSeek-R1-Distill-Llama-70B. Despite math-focused training, Light-R1-14B-DS demonstrates strong cross-domain generalization. Light-R1 represents a significant advancement in making sophisticated reasoning models more accessible and implementable in real-world applications. Our models, training data and code have been made available at https://github.com/Qihoo360/Light-R1.

Key facts. Cited by 175 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 48 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

30 of 48 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
18 more
  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  4. 2025Process Reinforcement through Implicit Rewardscited by 504
  5. 2025LIMO: Less is More for Reasoningcited by 498
  6. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  7. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  9. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 166
  10. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  11. 2025LIMR: Less is More for RL Scalingcited by 139
  12. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  13. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  14. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 98
  15. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  16. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  17. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 65
  18. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 130; 2026: 45.

175 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  2. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  5. 2025Learning to Reason under Off-Policy Guidancecited by 281
  6. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  7. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  8. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  11. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  12. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 97
38 more
  1. 2025Efficient Reasoning Models: A Surveycited by 95
  2. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  4. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 87
  5. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 82
  6. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergycited by 77
  7. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  8. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  9. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  10. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  11. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  12. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  13. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  14. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 50
  15. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  16. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 44
  17. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  18. 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Modelscited by 39
  19. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  20. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  21. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  22. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoningcited by 31
  23. 2026Purified OPSD: On-Policy Self-Distillation Without Losing How to Thinkcited by 31
  24. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  25. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 29
  26. 2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 28
  27. 2025Stable Reinforcement Learning for Efficient Reasoningcited by 27
  28. 2025Table-R1: Inference-Time Scaling for Table Reasoningcited by 24
  29. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  30. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulationcited by 20
  31. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 20
  32. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
  33. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  34. 2025CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generationcited by 19
  35. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  36. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 18
  37. 2025Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Modelscited by 18
  38. 2025Reasoning-Table: Exploring Reinforcement Learning for Table Reasoningcited by 18

50 most cited of 175 All 175 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Models
  2. 2025Token-Hungry, Yet Precise: DeepSeek R1 Highlights the Need for Multi-Step Reasoning Over Speed in MATH
  3. 2025Efficient Long CoT Reasoning in Small Language Models
  4. 2026Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning
  5. 2025Learning to Reason: Training LLMs with GPT-OSS or DeepSeek R1 Reasoning Traces
  6. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
  7. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
  8. 2025One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL