mlsanityData to 7 Oct 2026

JustRL: Scaling a 1.5B LLM with a Simple RL Recipe

arXiv 2512.16649 ↗First posted 18 Dec 2025cs.CLcited by 57

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in reinforcement learning for large language models have converged on increasing complexity: multi-stage training pipelines, dynamic hyperparameter schedules, and curriculum learning strategies. This raises a fundamental question: Is this complexity necessary? We present JustRL, a minimal approach using single-stage training with fixed hyperparameters that achieves state-of-the-art performance on two 1.5B reasoning models (54.9% and 64.3% average accuracy across nine mathematical benchmarks) while using 2× less compute than sophisticated approaches. The same hyperparameters transfer across both models without tuning, and training exhibits smooth, monotonic improvement over 4,000+ steps without the collapses or plateaus that typically motivate interventions. Critically, ablations reveal that adding ``standard tricks'' like explicit length penalties and robust verifiers may degrade performance by collapsing exploration. These results suggest that the field may be adding complexity to solve problems that disappear with a stable, scaled-up baseline. We release our models and code to establish a simple, validated baseline for the community.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

15 of 20 references.

References · 15

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 275
  8. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  9. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 162
  10. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  11. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 53
  12. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 39
3 more
  1. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 34
  2. 2025FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Modelscited by 26
  3. 2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 56.

57 papers.

Sort:

By area:

  1. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  2. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  3. 2025Evaluating Parameter Efficient Methods for RLVRcited by 12
  4. 2026Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillationcited by 10
  5. 2026STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokenscited by 9
  6. 2026OPRD: On-Policy Representation Distillationcited by 9
  7. 2026KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidancecited by 7
  8. 2026On-Policy Distillation with Best-of-N Teacher Rollout Selectioncited by 7
  9. 2026Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rolloutscited by 7
  10. 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreementcited by 7
  11. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-trainingcited by 6
  12. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 5
38 more
  1. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scalingcited by 5
  2. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  3. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewardscited by 3
  4. 2026Closing the Reflection Gap: A Free Calibration Bonus for Agentic RLcited by 3
  5. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  6. 2026Tail-Aware Top-k On-Policy Distillationcited by 3
  7. 2026Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Modelscited by 3
  8. 2026What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selectioncited by 3
  9. 2026Linear Dynamics in the RLVR Training of Large Language Modelscited by 2
  10. 2026Dissecting Failure Dynamics in Large Language Model Reasoningcited by 2
  11. 2026Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoningcited by 2
  12. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 2
  13. 2026Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillationcited by 2
  14. 2026Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoningcited by 1
  15. 2026LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Modelscited by 1
  16. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  17. 2026Fine-Tuning Small Reasoning Models for Quantum Field Theorycited by 1
  18. 2026Spectral Rewiring for Exploration, Purification, and Model Mergingcited by 1
  19. 2026Learning When to Trust via Selective Context Preference Optimizationcited by 1
  20. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progresscited by 1
  21. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  22. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  23. 2026Data-Free On-Policy Distillation: How Far Can We Go Without External Data?cited by 1
  24. 20261% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillationcited by 1
  25. 2026LastOPD: Taming Collapse in Latent On-Policy Distillationcited by 1
  26. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 0
  27. 2026Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoningcited by 0
  28. 2026Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoningcited by 0
  29. 2026Extreme Region Policy Distillationcited by 0
  30. 2026AdaStop: Cost-Aware Early Stopping for DNN Test Selectioncited by 0
  31. 2026ISO: An RLVR-Native Optimization Stackcited by 0
  32. 2026AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learningcited by 0
  33. 2026Constraint-First Reasoning: A Training-Free Protocol for Exploiting Answer-Space Constraints in Mathematical Problem Solvingcited by 0
  34. 2026TV-Regulated OPD: Direction Matters in On-Policy Distillationcited by 0
  35. 2026Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillationcited by 0
  36. 2026Reinforcement Learning with Verifiable Rewards for Small Search Agentscited by 0
  37. 2026Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?cited by 0
  38. 2026Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoderscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025TTRL: Test-Time Reinforcement Learning
  3. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  4. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models
  5. 2025SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  7. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  8. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling