mlsanityData to 7 Oct 2026

Scaling Reasoning Tokens via RL and Parallel Thinking: Evidence From Competitive Programming

arXiv 2604.01302 ↗First posted 1 Apr 2026cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We study how to scale reasoning token budgets for competitive programming through two complementary approaches: training-time reinforcement learning (RL) and test-time parallel thinking. During RL training, we observe an approximately log-linear relationship between validation accuracy and the average number of generated reasoning tokens over successive checkpoints, and show two ways to shift this training trajectory: verification RL warmup raises the starting point, while randomized clipping produces a steeper trend in the observed regime. As scaling single-generation reasoning during RL quickly becomes expensive under full attention, we introduce a multi-round parallel thinking pipeline that distributes the token budget across threads and rounds of generation, verification, and refinement. We train the model end-to-end on this pipeline to match the training objective to the test-time structure. Starting from Seed-OSS-36B, the full system with 16 threads and 16 rounds per thread matches the underlying RL model's oracle pass@16 at pass@1 using 7.6 million tokens per problem on average, and surpasses GPT-5-high on 456 hard competitive programming problems from AetherCode.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 50 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

37 of 50 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  8. 2025Process Reinforcement through Implicit Rewards3 Feb
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search6 Jun
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  12. 2023Let's Verify Step by Step31 May
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2025OpenAI GPT-5 System Cardcited by 910
12 more
  1. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  2. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  3. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  4. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  5. 2022Evolution through Large Modelscited by 131
  6. 2025Seed-Prover: Deep and Broad Reasoning for Automated Theorem Provingcited by 96
  7. 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoningcited by 84
  8. 2025BFS-Prover: Scalable Best-First Tree Search for LLM-based Automatic Theorem Provingcited by 82
  9. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 74
  10. 2017Multi-Task Learning for Contextual Banditscited by 47
  11. 2025Scaling up Multi-Turn Off-Policy RL and Multi-Agent Tree Search for LLM Step-Proverscited by 26
  12. 2025AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitionscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TMAS: Scaling Test-Time Compute via Multi-Agent Synergycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
  3. 2025Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
  4. 2026Batched Contextual Reinforcement: A Task-Scaling Law for Efficient Reasoning
  5. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning
  6. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  7. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers
  8. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL