mlsanityData to 7 Oct 2026

HardTests: Synthesizing High-Quality Test Cases for LLM Coding

arXiv 2505.24098 ↗First posted 30 May 2025cs.CLcited by 22

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Verifiers play a crucial role in large language model (LLM) reasoning, needed by post-training techniques such as reinforcement learning. However, reliable verifiers are hard to get for difficult coding problems, because a well-disguised wrong solution may only be detected by carefully human-written edge cases that are difficult to synthesize. To address this issue, we propose HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this pipeline, we curate a comprehensive competitive programming dataset HARDTESTS with 47k problems and synthetic high-quality tests. Compared with existing tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points higher and recall that is 17.5 percentage points higher when evaluating LLM-generated code. For harder problems, the improvement in precision can be as large as 40 points. HARDTESTS also proves to be more effective for model training, measured by downstream code generation performance. We will open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 36 of its 52 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

36 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset27 May
  2. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesis3 Feb
  3. 2024Scattered Forest Search: Smarter Code Space Exploration with LLMs22 Oct
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  5. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  12. 2025LIMO: Less is More for Reasoningcited by 488
19 more
  1. 2022CodeT: Code Generation with Generated Testscited by 369
  2. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  3. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  4. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  5. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  6. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  7. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  8. 2025LIMR: Less is More for RL Scalingcited by 136
  9. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  10. 2025Competitive Programming with Large Reasoning Modelscited by 115
  11. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  12. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  13. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  14. 2024TestGenEval: A Real World Unit Test Generation and Test Completion Benchmarkcited by 49
  15. 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 38
  16. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 32
  17. 2025COFFE: A Code Efficiency Benchmark for Code Generationcited by 18
  18. 2024TestBench: Evaluating Class-Level Test Case Generation Capability of Large Language Modelscited by 15
  19. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 16.

22 papers.

Sort:

By area:

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  2. 2025AutoCode: LLMs as Problem Setters for Competitive Programmingcited by 11
  3. 2025Klear-CodeTest: Scalable Test Case Generation for Code Reinforcement Learningcited by 9
  4. 2025Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learningcited by 7
  5. 2026CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutionscited by 6
  6. 2026Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learningcited by 6
  7. 2026EvolveCoder: Evolving Test Cases via Adversarial Verification for Code Reinforcement Learningcited by 5
  8. 2025How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspectivecited by 4
  9. 2026FrontierSmith: Synthesizing Open-Ended Coding Problems at Scalecited by 3
  10. 2026CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programmingcited by 2
  11. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensuscited by 1
  12. 2026VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generationcited by 1
10 more
  1. 2026Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programmingcited by 1
  2. 2026Learning Generalizable Behaviors for Terminal Agentscited by 1
  3. 2025NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generationcited by 0
  4. 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Testscited by 0
  5. 2026RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learningcited by 0
  6. 2026Navigating the Clutter: Waypoint-Based Bi-Level Planning for Multi-Robot Systemscited by 0
  7. 2026D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discoverycited by 0
  8. 2026The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithmscited by 0
  9. 2026Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shapingcited by 0
  10. 2026Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  2. 2025CodeContests+: High-Quality Test Case Generation for Competitive Programming
  3. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
  4. 2025Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
  5. 2025Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
  6. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation
  7. 2026CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation
  8. 2025QueST: Incentivizing LLMs to Generate Difficult Problems