mlsanityData to 14 Oct 2026

CodeContests+: High-Quality Test Case Generation for Competitive Programming

arXiv 2506.05817 ↗First posted 6 Jun 2025cs.SE cs.CLcited by 46

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Competitive programming, due to its high reasoning difficulty and precise correctness feedback, has become a key task for both training and evaluating the reasoning capabilities of large language models (LLMs). However, while a large amount of public problem data, such as problem statements and solutions, is available, the test cases of these problems are often difficult to obtain. Therefore, test case generation is a necessary task for building large-scale datasets, and the quality of the test cases directly determines the accuracy of the evaluation. In this paper, we introduce an LLM-based agent system that creates high-quality test cases for competitive programming problems. We apply this system to the CodeContests dataset and propose a new version with improved test cases, named CodeContests+. We evaluated the quality of test cases in CodeContestsPlus. First, we used 1.72 million submissions with pass/fail labels to examine the accuracy of these test cases in evaluation. The results indicated that CodeContests+ achieves significantly higher accuracy than CodeContests, particularly with a notably higher True Positive Rate (TPR). Subsequently, our experiments in LLM Reinforcement Learning (RL) further confirmed that improvements in test case quality yield considerable advantages for RL.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 12 of its 19 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

12 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  2. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 10

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2021Measuring Coding Challenge Competence With APPScited by 902
  7. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  8. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  9. 2024Can Language Models Solve Olympiad Programming?cited by 72
  10. 2024FullStack Bench: Evaluating LLMs as Full Stack Coderscited by 58

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 33.

46 papers.

Sort:

By area:

  1. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 33
  2. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  3. 2025AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitionscited by 11
  4. 2025Learning to Generate Unit Test via Adversarial Reinforcement Learningcited by 11
  5. 2025AutoCode: LLMs as Problem Setters for Competitive Programmingcited by 11
  6. 2025Klear-CodeTest: Scalable Test Case Generation for Code Reinforcement Learningcited by 9
  7. 2026CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutionscited by 8
  8. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  9. 2026TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generationcited by 5
  10. 2025How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspectivecited by 4
  11. 2025LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?cited by 4
  12. 2026GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learningcited by 4
34 more
  1. 2026SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineeringcited by 4
  2. 2026ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenascited by 3
  3. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 3
  4. 2026VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generationcited by 3
  5. 2026FrontierSmith: Synthesizing Open-Ended Coding Problems at Scalecited by 3
  6. 2026Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolutioncited by 3
  7. 2025VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Modelscited by 2
  8. 2025HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learningcited by 2
  9. 2026Learning Generative Selection for Best-of-Ncited by 2
  10. 2026RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learningcited by 2
  11. 2026Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RLcited by 2
  12. 2026Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programmingcited by 2
  13. 2025UniCode: Augmenting Evaluation for Code Reasoningcited by 1
  14. 2025EvoSyn: Generalizable Evolutionary Data Synthesis for Verifiable Learningcited by 1
  15. 2026CSSG: Measuring Code Similarity with Semantic Graphscited by 1
  16. 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Testscited by 1
  17. 2026Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programmingcited by 1
  18. 2026D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discoverycited by 1
  19. 2026ARIADNE: Agentic Reward-Informed Adaptive Decision Exploration via Blackboard-Driven MCTS for Competitive Program Generationcited by 1
  20. 2026Debate Helps Weak Judges Reward Stronger Modelscited by 1
  21. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalizationcited by 1
  22. 2025TGPR: Tree-Guided Policy Refinement for Robust Self-Debugging of LLMscited by 0
  23. 2026MAS-Algorithm: A Workflow for Solving Algorithmic Programming Problems with a Multi-Agent Systemcited by 0
  24. 2026Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuningcited by 0
  25. 2026The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithmscited by 0
  26. 2026MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validationcited by 0
  27. 2026Reinforcement Learning for Code Optimizationcited by 0
  28. 2026Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catchcited by 0
  29. 2026Task Specialization Fine-Tuning for Contextual Reinforcement Learningcited by 0
  30. 2026Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shapingcited by 0
  31. 2026InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Informationcited by 0
  32. 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMscited by 0
  33. 2026DivAlgo: A systematic evaluation of algorithmic diversity for code generation modelscited by 0
  34. 2026LintEval: Benchmarking LLMs on Linter-Guided File Refactoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation
  2. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Coding
  3. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  4. 2025Evaluating and Improving Large Language Models for Competitive Program Generation
  5. 2025Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
  6. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancement
  7. 2026Beyond the Verdict: Agentic Code Judging with Verifiable Feedback
  8. 2023Competition-Level Problems are Effective LLM Evaluators