# FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair

arXiv: https://arxiv.org/abs/2307.00012 · HTML: https://mlsanity.com/paper/2307.00012/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 92 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [CodeScore: Evaluating Code Generation by Learning Code Execution](https://mlsanity.com/paper/2301.09043/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [A Twofold Siamese Network for Real-Time Object Tracking](https://mlsanity.com/paper/1802.08817/) (2018)
- [Applying CodeBERT for Automated Program Repair of Java Simple Bugs](https://mlsanity.com/paper/2103.11626/) (2021)
- [Improving ChatGPT Prompt for Code Generation](https://mlsanity.com/paper/2305.08360/) (2023)
- [An Empirical Comparison of Pre-Trained Models of Source Code](https://mlsanity.com/paper/2302.04026/) (2023)
- [Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests](https://mlsanity.com/paper/2112.12331/) (2021)
- [Understanding Flaky Tests: The Developer's Perspective](https://mlsanity.com/paper/1907.01466/) (2019)
- [A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests](https://mlsanity.com/paper/2112.04919/) (2021)
- [What is the Vocabulary of Flaky Tests? An Extended Replication](https://mlsanity.com/paper/2103.12670/) (2021)
- [A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It](https://mlsanity.com/paper/2203.00483/) (2022)
- [Practical Flaky Test Prediction using Common Code Evolution and Test History Data](https://mlsanity.com/paper/2302.09330/) (2023)
- [FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. A Case Study on Mutation Testing and Program Repair](https://mlsanity.com/paper/1912.03197/) (2019)
- [An Empirical Study of Flaky Tests in JavaScript](https://mlsanity.com/paper/2207.01047/) (2022)
- [Time-based Repair for Asynchronous Wait Flaky Tests in Web Testing](https://mlsanity.com/paper/2305.08592/) (2023)

## Built on by (most cited first, 6 of 6)

- [Large Language Models for Software Testing: A Research Roadmap](https://mlsanity.com/paper/2509.25043/) (2025)
- [Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models](https://mlsanity.com/paper/2601.22264/) (2026)
- [FlaKat: A Machine Learning-Based Categorization Framework for Flaky Tests](https://mlsanity.com/paper/2403.01003/) (2024)
- [An Empirical Study of Sample Selection Strategies for Large Language Model Repair](https://mlsanity.com/paper/2510.20428/) (2025)
- [FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale](https://mlsanity.com/paper/2511.14002/) (2025)
- [Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software](https://mlsanity.com/paper/2603.09029/) (2026)

## Abstract (arXiv)

Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test, causing confusion and wasting development effort. While machine learning models have been used to predict flakiness and its root causes, there is much less work on providing support to fix the problem. To address this gap, in this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis. We do this for a subset of flaky tests where the root cause of flakiness is in the test itself and not in the production code. One key idea is to guide the repair process with additional knowledge about the test's flakiness in the form of its predicted fix category. Thus, we first propose a framework that automatically generates labeled datasets for 13 fix categories and trains models to predict the fix category of a flaky test by analyzing the test code only. Our experimental results using code models and few-shot learning show that we can correctly predict most of the fix categories. To show the usefulness of such fix category labels for automatically repairing flakiness, we augment the prompts of GPT-3.5 Turbo, a Large Language Model (LLM), with such extra knowledge to request repair suggestions. The results show that our suggested fix category labels, complemented with in-context learning, significantly enhance the capability of GPT-3.5 Turbo in generating fixes for flaky tests. Based on the execution and analysis of a sample of GPT-repaired flaky tests, we estimate that a large percentage of such repairs (roughly between 51% and 83%) can be expected to pass. For the failing repaired tests, on average, 16% of the test code needs to be further changed for them to pass.