mlsanityData to 13 Oct 2026

exLong: Generating Exceptional Behavior Tests with Large Language Models

arXiv 2405.14619 ↗First posted 23 May 2024cs.SE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Many popular programming languages, including C#, Java, and Python, support exceptions. Exceptions are thrown during program execution if an unwanted event happens, e.g., a method is invoked with an illegal argument value. Software developers write exceptional behavior tests (EBTs) to check that their code detects unwanted events and throws appropriate exceptions. Prior research studies have shown the importance of EBTs, but those studies also highlighted that developers put most of their efforts on "happy paths", e.g., paths without unwanted events. To help developers fill the gap, we present the first framework, dubbed exLong, that automatically generates EBTs. exLong is a large language model instruction fine-tuned from CodeLlama and embeds reasoning about traces that lead to throw statements, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. We compare exLong with the state-of-the-art models for test generation (CAT-LM) and one of the strongest foundation models (GPT-4o), as well as with analysis-based tools for test generation (Randoop and EvoSuite). Our results show that exLong outperforms existing models and tools. Furthermore, we contributed several pull requests to open-source projects and 23 EBTs generated by exLong were already accepted.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 28 of its 69 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

28 of 69 references.

References · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,130
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 884
  7. 2023StarCoder: may the source be with you!cited by 797
  8. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 651
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 475
  10. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  11. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 184
16 more
  1. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 164
  2. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  3. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  4. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 64
  5. 2021TOGA: A Neural Method for Test Oracle Generationcited by 57
  6. 2023Using Large Language Models to Generate JUnit Tests: An Empirical Studycited by 53
  7. 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
  8. 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
  9. 2020On Learning Meaningful Assert Statements for Unit Test Casescited by 42
  10. 2020Learning to Update Natural Language Comments Based on Code Changescited by 31
  11. 2023Learning Deep Semantics for Test Completioncited by 31
  12. 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30
  13. 2023CAT-LM: Training Language Models on Aligned Code And Testscited by 28
  14. 2023Multilingual Code Co-Evolution Using Large Language Modelscited by 9
  15. 2021Using Relative Lines of Code to Guide Automated Test Generation for Pythoncited by 3
  16. 2021Assessing Exception Handling Testing Practices in Open-Source Librariescited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Navigating the Labyrinth: Path-Sensitive Unit Test Generation with Large Language Modelscited by 1
  2. 2026Multimodal QUD: Inquisitive Questions from Scientific Figurescited by 0
  3. 2026STAB: Specification-driven Testing for Algorithmic Bottleneckscited by 0
  4. 2026Retrofitting Code Using LLMs to Support Exceptional Behaviorcited by 0