mlsanityData to 14 Oct 2026

Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

arXiv 2506.12278 ↗First posted 13 Jun 2025cs.SE cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation. TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform. It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation. We provide a comprehensive assessment of 19 state-of-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

14 of 21 references.

References · 14

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  8. 2021Measuring Coding Challenge Competence With APPScited by 902
  9. 2022CodeT: Code Generation with Generated Testscited by 384
  10. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  11. 2025ACECODER: Acing Coder RL via Automated Test-Case Synthesiscited by 93
  12. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
2 more
  1. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 39
  2. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  2. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 13
  3. 2025How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspectivecited by 4
  4. 2026ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generationcited by 2
  5. 2026Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programmingcited by 2
  6. 2025Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Frameworkcited by 1
  7. 2025UniCode: Augmenting Evaluation for Code Reasoningcited by 1
  8. 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Testscited by 1
  9. 2026Agents with Feelings? Personality and Emotion in Multi-Agent Software Teamscited by 0
  10. 2026Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catchcited by 0