mlsanityData to 7 Oct 2026

Kimi k1.5: Scaling Reinforcement Learning with LLMs

arXiv 2501.12599 ↗First posted 22 Jan 2025cs.AI cs.LGcited by 1,193

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).

Key facts. Cited by 1,193 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 64 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

53 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2023Let's Verify Step by Step31 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
38 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  13. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  14. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  15. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  16. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  17. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  18. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  19. 2023Scaling Data-Constrained Language Modelscited by 391
  20. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  21. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  22. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  23. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  24. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  25. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  26. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  27. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  28. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  29. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  30. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  31. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  32. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  33. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 151
  34. 2024LLM Critics Help Catch LLM Bugscited by 138
  35. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  36. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  37. 2024Critique-out-Loud Reward Modelscited by 105
  38. 2020Mirror Descent Policy Optimizationcited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 770; 2026: 423.

1,193 papers.

Sort:

By area:

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  4. 2025Kimi K2: Open Agentic Intelligencecited by 523
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  6. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  7. 2025Process Reinforcement through Implicit Rewardscited by 478
  8. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  9. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  10. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  11. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  12. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
38 more
  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  3. 2025Muon is Scalable for LLM Trainingcited by 381
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  5. 2025Kimi-VL Technical Reportcited by 381
  6. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  7. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  8. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  9. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  10. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  11. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  12. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  13. 2026Reinforcement Learning via Self-Distillationcited by 285
  14. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  15. 2025Learning to Reason under Off-Policy Guidancecited by 266
  16. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  17. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  18. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  19. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  20. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  21. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  22. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  23. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  24. 2025Kimi-Audio Technical Reportcited by 210
  25. 2025Learning to Reason without External Rewardscited by 199
  26. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  27. 2025Deep Think with Confidencecited by 199
  28. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  29. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  30. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  31. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  32. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  33. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  34. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  35. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  36. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 173
  37. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  38. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169

50 most cited of 1,193 All 1,193 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  3. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
  4. 2024Reinforcement Learning Enhanced LLMs: A Survey
  5. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
  6. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMs
  7. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  8. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models