mlsanityData to 7 Oct 2026

Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests

arXiv 2607.22883 ↗First posted 24 Jul 2026cs.SE cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it. Our analysis reveals that prompting LLMs with buggy code has a severe, twofold impact: it significantly increases "misguided tests" that assert incorrect behavior while simultaneously suppressing the generation of effective, bug-finding tests. We further corroborate this effect from a model-internal perspective, showing that buggy code skews LLMs' preference toward tests that assert the same erroneous behavior. To counter this, we introduce and validate a specification-based unit test generation paradigm that replaces the code under test in the prompt with an LLM-generated specification docstring. Our results show that this paradigm effectively reduces misguided tests while substantially increasing effective tests, improves multi-round, feedback-driven test generation pipelines, and remains applicable to both buggy and bug-free code. Overall, these results suggest that specification-based prompting is a promising strategy for mitigating misguidance from buggy code in LLM-generated unit tests.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 75 references that are in the map. First posted on arXiv 24 Jul 2026.

Builds on

40 of 75 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  8. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  9. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  10. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  11. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  12. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
28 more
  1. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  2. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 94
  3. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 73
  4. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  5. 2021TOGA: A Neural Method for Test Oracle Generationcited by 57
  6. 2024Exploring and Unleashing the Power of Large Language Models in Automated Code Translationcited by 57
  7. 2023Using Large Language Models to Generate JUnit Tests: An Empirical Studycited by 53
  8. 2023Effective Test Generation Using Pre-trained Large Language Models and Mutation Testingcited by 51
  9. 2022Pynguin: Automated Unit Test Generation for Pythoncited by 49
  10. 2024HITS: High-coverage LLM-based Unit Test Generation via Method Slicingcited by 37
  11. 2024CoverUp: Effective High Coverage Test Generation for Pythoncited by 36
  12. 2023A3Test: Assertion-Augmented Automated Test Case Generationcited by 33
  13. 2024On the Evaluation of Large Language Models in Unit Test Generationcited by 33
  14. 2023ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generationcited by 32
  15. 2023CAT-LM: Training Language Models on Aligned Code And Testscited by 28
  16. 2023Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Promptingcited by 23
  17. 2024TOGLL: Correct and Strong Test Oracle Generation with LLMscited by 23
  18. 2024ThinkRepair: Self-Directed Automated Program Repaircited by 22
  19. 2024A Large-scale Empirical Study on Fine-tuning Large Language Models for Unit Testingcited by 15
  20. 2024LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programscited by 14
  21. 2024Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generationcited by 10
  22. 2024ASTER: Natural and Multi-language Unit Test Generation with LLMscited by 9
  23. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 9
  24. 2025TestForge: Feedback-Driven, Agentic Test Suite Generationcited by 8
  25. 2023Domain Adaptation for Code Model-based Unit Test Case Generationcited by 7
  26. 2025Test Wars: A Comparative Study of SBST, Symbolic Execution, and LLM-Based Approaches to Unit Test Generationcited by 7
  27. 2024Parameter-Efficient Fine-Tuning of Large Language Models for Unit Test Generation: An Empirical Studycited by 3
  28. 2024Design choices made by LLM-based test generators prevent them from finding bugscited by 2

Built on by

Not cited yet.