mlsanityData to 14 Oct 2026

S*: Test Time Scaling for Code Generation

arXiv 2502.14382 ↗First posted 20 Feb 2025cs.LG cs.AIcited by 98

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S*, the first hybrid test-time scaling framework that substantially improves the coverage and selection accuracy of generated code. S* extends the existing parallel scaling paradigm with sequential scaling to push performance boundaries. It further leverages a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison, combined with execution-grounded information to robustly identify correct solutions. We evaluate across 12 Large Language Models and Large Reasoning Model and show: (1) S* consistently improves performance across model families and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables non-reasoning models to surpass reasoning models - GPT-4o-mini with S* outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S* achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be available under https://github.com/NovaSky-AI/SkyThought.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 39 of its 57 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

39 of 57 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025CodeMonkeys: Scaling Test-Time Compute for Software Engineering24 Jan
  2. 2024Qwen2.5-Coder Technical Report18 Sep
  3. 2024RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation15 Sep
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  5. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency29 Sep
  6. 2023Teaching Large Language Models to Self-Debug11 Apr
  7. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
20 more
  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  2. 2021Measuring Coding Challenge Competence With APPScited by 902
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2022CodeT: Code Generation with Generated Testscited by 384
  7. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  8. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  9. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  10. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  11. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  12. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
  13. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  14. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 64
  15. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 60
  16. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  17. 2024The Limits of Inference Scaling Through Resamplingcited by 53
  18. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  19. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  20. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 47; 2026: 51.

98 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  4. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 94
  5. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 61
  6. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  7. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  8. 2025SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoningcited by 41
  9. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 29
  10. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 24
  11. 2025ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generationcited by 22
  12. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
38 more
  1. 2025Classical Planning with LLM-Generated Heuristics: Challenging the State of the Art with Python Codecited by 18
  2. 2025Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Studycited by 16
  3. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  5. 2025MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancementcited by 14
  6. 2025SSRL: Self-Search Reinforcement Learningcited by 14
  7. 2025GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMscited by 13
  8. 2025Co-PatcheR: Collaborative Software Patching with Component(s)-specific Small Reasoning Modelscited by 12
  9. 2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
  10. 2025Long Is More Important Than Difficult for Training Reasoning Modelscited by 11
  11. 2025SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Modelscited by 10
  12. 2025First Finish Search: Efficient Test-Time Scaling in Large Language Modelscited by 9
  13. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  14. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  15. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  16. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  17. 2025Scaling Textual Gradients via Sampling-Based Momentumcited by 6
  18. 2025Investigating Test Overfitting on SWE-benchcited by 6
  19. 2025Test-Time Scaling of Reasoning Models for Machine Translationcited by 5
  20. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  21. 2026SEMAG: Self-Evolutionary Multi-Agent Code Generationcited by 5
  22. 2026SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Testscited by 5
  23. 2025SSR: Speculative Parallel Scaling Reasoning in Test-timecited by 4
  24. 2025DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problemscited by 4
  25. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  26. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?cited by 4
  27. 2026Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Modelscited by 4
  28. 2025Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignmentcited by 3
  29. 2025ATTS: Asynchronous Test-Time Scaling via Conformal Predictioncited by 3
  30. 2025R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learningcited by 3
  31. 2025InfCode-C++: Intent-Guided Semantic Retrieval and AST-Structured Search for C++ Issue Resolutioncited by 3
  32. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3
  33. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  34. 2026FunPRM: Function-as-Step Process Reward Model with Meta Reward Correction for Code Generationcited by 3
  35. 2026A Theoretical Analysis of Test-Driven Code Generationcited by 3
  36. 2026GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedbackcited by 3
  37. 2026Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generationcited by 3
  38. 2026Visual Verification Enables Inference-time Steering and Autonomous Policy Improvementcited by 3

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Z1: Efficient Test-time Scaling with Code
  2. 2026Code Generation by Differential Test Time Scaling
  3. 2025Dynamic Scaling of Unit Tests for Code Reward Modeling
  4. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
  5. 2025Are We Scaling the Right Thing? A System Perspective on Test-Time Scaling
  6. 2026PaT: Planning-after-Trial for Efficient Test-Time Code Generation
  7. 2025Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models
  8. 2026ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning