# Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests

arXiv: https://arxiv.org/abs/2112.12331 · HTML: https://mlsanity.com/paper/2112.12331/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2021.

## Built on by (most cited first, 11 of 11)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [NaviQAte: Functionality-Guided Web Application Navigation](https://mlsanity.com/paper/2409.10741/) (2024)
- [Can GPT-4 Replicate Empirical Software Engineering Research?](https://mlsanity.com/paper/2310.01727/) (2023)
- [FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair](https://mlsanity.com/paper/2307.00012/) (2023)
- [An Analysis of LLM Fine-Tuning and Few-Shot Learning for Flaky Test Detection and Classification](https://mlsanity.com/paper/2502.02715/) (2025)
- [Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models](https://mlsanity.com/paper/2601.22264/) (2026)
- [FlaKat: A Machine Learning-Based Categorization Framework for Flaky Tests](https://mlsanity.com/paper/2403.01003/) (2024)
- [A Match Made in Heaven? AI-driven Matching of Vulnerabilities and Security Unit Tests](https://mlsanity.com/paper/2502.03365/) (2025)
- [FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale](https://mlsanity.com/paper/2511.14002/) (2025)
- [Explicating Tacit Regulatory Knowledge from LLMs to Auto-Formalize Requirements for Compliance Test Case Generation](https://mlsanity.com/paper/2601.09762/) (2026)
- [Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software](https://mlsanity.com/paper/2603.09029/) (2026)

## Abstract (arXiv)

Software testing assures that code changes do not adversely affect existing functionality. However, a test case can be flaky, i.e., passing and failing across executions, even for the same version of the source code. Flaky test cases introduce overhead to software development as they can lead to unnecessary attempts to debug production or testing code. The state-of-the-art ML-based flaky test case predictors rely on pre-defined sets of features that are either project-specific, require access to production code, which is not always available to software test engineers. Therefore, in this paper, we propose Flakify, a black-box, language model-based predictor for flaky test cases. Flakify relies exclusively on the source code of test cases, thus not requiring to (a) access to production code (black-box), (b) rerun test cases, (c) pre-define features. To this end, we employed CodeBERT, a pre-trained language model, and fine-tuned it to predict flaky test cases using the source code of test cases. We evaluated Flakify on two publicly available datasets (FlakeFlagger and IDoFT) for flaky test cases and compared our technique with the FlakeFlagger approach using two different evaluation procedures: cross-validation and per-project validation. Flakify achieved high F1-scores on both datasets using cross-validation and per-project validation, and surpassed FlakeFlagger by 10 and 18 percentage points in terms of precision and recall, respectively, when evaluated on the FlakeFlagger dataset, thus reducing the cost bound to be wasted on unnecessarily debugging test cases and production code by the same percentages. Flakify also achieved significantly higher prediction results when used to predict test cases on new projects, suggesting better generalizability over FlakeFlagger. Our results further show that a black-box version of FlakeFlagger is not a viable option for predicting flaky test cases.