mlsanityData to 7 Oct 2026

TESTEVAL: Benchmarking Large Language Models for Test Case Generation

arXiv 2406.04531 ↗First posted 6 Jun 2024cs.SEcited by 38

Not on the map. Why?

Abstract (arXiv)

Testing plays a crucial role in the software development cycle, enabling the detection of bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers need to write code snippets that execute the program under test. Recently, researchers have recognized the potential of large language models (LLMs) in software testing. However, there remains a lack of fair comparisons between different LLMs in terms of test case generation capabilities. In this paper, we propose TESTEVAL, a novel benchmark for test case generation with LLMs. We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage. We further evaluate sixteen popular LLMs, including both commercial and open-source ones, on TESTEVAL. We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs, indicating a lack of ability to comprehend program logic and execution paths. We have open-sourced our dataset and benchmark pipelines at https://github.com/LLM4SoftwareTesting/TestEval.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Jun 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 20; 2026: 15.

38 papers.

Sort:

By area:

  1. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  2. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 32
  3. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Codingcited by 22
  4. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  5. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  6. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  7. 2025Rethinking Verification for LLM Code Generation: From Generation to Testingcited by 12
  8. 2025Learning to Generate Unit Test via Adversarial Reinforcement Learningcited by 11
  9. 2026DevOps-Gym: Benchmarking AI Agents in Software DevOps Cyclecited by 9
  10. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoningcited by 8
  11. 2025Benchmarking LLMs for Unit Test Generation from Real-World Functionscited by 8
  12. 2025CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verificationcited by 7
26 more
  1. 2025Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Modelscited by 7
  2. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
  3. 2026TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generationcited by 5
  4. 2025Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generationcited by 4
  5. 2025LogiCase: Effective Test Case Generation from Logical Description in Competitive Programmingcited by 4
  6. 2025Past, Present, and Future of Bug Tracking in the Generative AI Eracited by 4
  7. 2025Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesiscited by 4
  8. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cyclecited by 4
  9. 2024Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropycited by 3
  10. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  11. 2025ContractEval: A Benchmark for Evaluating Contract-Satisfying Assertions in Code Generationcited by 2
  12. 2026RM -RF: Reward Model for Run-Free Unit Test Evaluationcited by 2
  13. 2026Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steeringcited by 2
  14. 2025A Systematic Approach for Assessing Large Language Models' Test Case Generation Capabilitycited by 1
  15. 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
  16. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensuscited by 1
  17. 2026SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generationcited by 1
  18. 2025SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMscited by 0
  19. 2025Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Modelscited by 0
  20. 2026TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenancecited by 0
  21. 2026An empirical study of LoRA-based fine-tuning of large language models for automated test case generationcited by 0
  22. 2026Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automationcited by 0
  23. 2026Rethinking Code Performance Benchmarks for LLMscited by 0
  24. 2026Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)cited by 0
  25. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmarkcited by 0
  26. 2026Characterizing Overconfident Failure in LLM-Based Code Generationcited by 0