mlsanityData to 7 Oct 2026

Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning

arXiv 2510.21398 ↗First posted 24 Oct 2025cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Test-time scaling methods have seen a rapid increase in popularity for its computational efficiency and parameter-independent training to improve reasoning performance on Large Language Models. One such method is called budget forcing, a decoding intervention strategy which allocates extra compute budget for thinking and elicits the inherent self-correcting behavior of the model. However, this relies on supervised fine-tuning (SFT) on long-context reasoning traces which causes performance degradation on smaller models due to verbose responses. For this reason, we offer a framework integrating reinforcement learning (RL) to improve token efficiency and boost the performance of a 1.5B model for mathematical reasoning. We demonstrate this using only 1.5K training samples and found that our SFT+RL model performed better on the GSM8K dataset with varying compute budgets. Our main findings showed an overall higher accuracy while significantly reducing its token usage by over 40% compared to the SFT model, revealing how RL can recover the losses due to long-context training and altogether improving performance in mathematical reasoning.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 51 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

40 of 51 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning18 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  7. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 30

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  12. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
18 more
  1. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  5. 2025LIMO: Less is More for Reasoningcited by 488
  6. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  7. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  8. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  9. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  10. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  11. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  12. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  13. 2025SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scalingcited by 54
  14. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  15. 2025Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?cited by 21
  16. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoningcited by 13
  17. 2025Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcementcited by 13
  18. 2025Long Is More Important Than Difficult for Training Reasoning Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning Adaptive Reasoning Paths for Efficient Visual Reasoningcited by 5
  2. 2026Asymmetric Capacity Allocation in Self-Refinement Pipelinescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens
  2. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  3. 2025A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning
  4. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  5. 2025Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem
  6. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
  7. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
  8. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning