mlsanityData to 7 Oct 2026

ACECODER: Acing Coder RL via Automated Test-Case Synthesis

arXiv 2502.01718 ↗First posted 3 Feb 2025cs.SE cs.AI cs.CLcited by 89

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Most progress in recent coder models has been driven by supervised fine-tuning (SFT), while the potential of reinforcement learning (RL) remains largely unexplored, primarily due to the lack of reliable reward data/model in the code domain. In this paper, we address this challenge by leveraging automated large-scale test-case synthesis to enhance code model training. Specifically, we design a pipeline that generates extensive (question, test-cases) pairs from existing code data. Using these test cases, we construct preference pairs based on pass rates over sampled programs to train reward models with Bradley-Terry loss. It shows an average of 10-point improvement for Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5. Furthermore, we conduct reinforcement learning with both reward models and test-case pass rewards, leading to consistent improvements across HumanEval, MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the R1-style training to start from Qwen2.5-Coder-base directly and show that our RL training can improve model on HumanEval-plus by over 25% and MBPP-plus by 6% for merely 80 optimization steps. We believe our results highlight the huge potential of reinforcement learning in coder models.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 37 of its 49 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

37 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning2 Oct
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  3. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement22 Feb
  4. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback2 Feb
  5. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency29 Sep

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
20 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  6. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  7. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  10. 2022CodeT: Code Generation with Generated Testscited by 369
  11. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  12. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  13. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  14. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  15. 2024Evaluating Language Models for Efficient Code Generationcited by 69
  16. 2023Clover: Closed-Loop Verifiable Code Generationcited by 62
  17. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  18. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  19. 2023Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generationcited by 21
  20. 2024DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 48; 2026: 41.

89 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  4. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  5. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  6. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  7. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  8. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  9. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  10. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 49
  11. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 47
  12. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
38 more
  1. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
  2. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  3. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  4. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewardscited by 25
  5. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  6. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  7. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  8. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Codingcited by 22
  9. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 18
  10. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  11. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  12. 2025Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Teamcited by 13
  13. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  14. 2025Rethinking Verification for LLM Code Generation: From Generation to Testingcited by 12
  15. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 12
  16. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  17. 2025Rethinking the Evaluation of Secure Code Generationcited by 10
  18. 2025Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposurecited by 10
  19. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoningcited by 8
  20. 2025Graph-Reward-SQL: Execution-Free Reinforcement Learning for Text-to-SQL via Graph Matching and Stepwise Rewardcited by 7
  21. 2025Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learningcited by 7
  22. 2026Harnessing LLM Agents with Skill Programscited by 7
  23. 2026Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learningcited by 6
  24. 2025RAST: Reasoning Activation in LLMs via Small-model Transfercited by 5
  25. 2025Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environmentcited by 5
  26. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  27. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  28. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 5
  29. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  30. 2025AgentPack: A Dataset of Code Changes, Co-Authored by Agents and Humanscited by 4
  31. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  32. 2025Zero Reinforcement Learning Towards General Domainscited by 4
  33. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  34. 2026AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineeringcited by 4
  35. 2026Latent Reasoning with Normalizing Flowscited by 4
  36. 2025CTTS: Collective Test-Time Scalingcited by 3
  37. 2025Learning to Reason as Action Abstractions with Scalable Mid-Training RLcited by 3
  38. 2025VisCoder2: Building Multi-Language Visualization Coding Agentscited by 3

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
  2. 2026Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
  3. 2025R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning
  4. 2026Improving Small Language Models for Code Generation with Reinforcement Learning from Verification Feedback
  5. 2026When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study
  6. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
  7. 2026Reinforcement Learning for Code Optimization
  8. 2025CodeContests+: High-Quality Test Case Generation for Competitive Programming