# FlakeRanker: Automated Identification and Prioritization of Flaky Job Failure Categories

arXiv: https://arxiv.org/abs/2503.12312 · HTML: https://mlsanity.com/paper/2503.12312/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2025.

## Built on by (most cited first, 1 of 1)

- [Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models](https://mlsanity.com/paper/2601.22264/) (2026)

## Abstract (arXiv)

This document presents the artifact associated with the ICSE SEIP 25 paper titled On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories. The original paper identifies and analyzes 46 distinct categories of flaky job failures that developers encounter, using Recency (R), Frequency (F), and Monetary (M) measures. In addition, it uses an RFM clustering model to identify and prioritize the most wasteful and persistent. The original paper only discusses the rankings and evolution of the top 20 categories in the results. This artifact contains (1) the regex and scripts used to automate the labeling process for RQ1, (2) complete analysis results, including the ranking of all 46 categories by cost in RQ2 and the evolution of these categories over time in RQ3, and (3) the RFM dataset and scripts used to create the RFM clustering model for prioritization in RQ4. In addition, we engineered the labeling tool and the RFM-based prioritization methodology in a command-line interface (CLI) called FLAKERANKER to facilitate reuse and repurposing in future studies.