# FlaKat: A Machine Learning-Based Categorization Framework for Flaky Tests

arXiv: https://arxiv.org/abs/2403.01003 · HTML: https://mlsanity.com/paper/2403.01003/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 51 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 11 of 11)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests](https://mlsanity.com/paper/2112.12331/) (2021)
- [Understanding Flaky Tests: The Developer's Perspective](https://mlsanity.com/paper/1907.01466/) (2019)
- [Adapting Neural Text Classification for Improved Software Categorization](https://mlsanity.com/paper/1806.01742/) (2018)
- [FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair](https://mlsanity.com/paper/2307.00012/) (2023)
- [A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests](https://mlsanity.com/paper/2112.04919/) (2021)
- [Practical Flaky Test Prediction using Common Code Evolution and Test History Data](https://mlsanity.com/paper/2302.09330/) (2023)
- [On the Effect of Instrumentation on Test Flakiness](https://mlsanity.com/paper/2303.09755/) (2023)
- [Debugging Flaky Tests using Spectrum-based Fault Localization](https://mlsanity.com/paper/2305.04735/) (2023)

## Abstract (arXiv)

Flaky tests can pass or fail non-deterministically, without alterations to a software system. Such tests are frequently encountered by developers and hinder the credibility of test suites. State-of-the-art research incorporates machine learning solutions into flaky test detection and achieves reasonably good accuracy. Moreover, the majority of automated flaky test repair solutions are designed for specific types of flaky tests. This research work proposes a novel categorization framework, called FlaKat, which uses machine-learning classifiers for fast and accurate prediction of the category of a given flaky test that reflects its root cause. Sampling techniques are applied to address the imbalance between flaky test categories in the International Dataset of Flaky Test (IDoFT). A new evaluation metric, called Flakiness Detection Capacity (FDC), is proposed for measuring the accuracy of classifiers from the perspective of information theory and provides proof for its effectiveness. The final FDC results are also in agreement with F1 score regarding which classifier yields the best flakiness classification.