# On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories

arXiv: https://arxiv.org/abs/2501.04976 · HTML: https://mlsanity.com/paper/2501.04976/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2025.

## Built on by (most cited first, 2 of 2)

- [Efficient Detection of Intermittent Job Failures Using Few-Shot Learning](https://mlsanity.com/paper/2507.04173/) (2025)
- [Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models](https://mlsanity.com/paper/2601.22264/) (2026)

## Abstract (arXiv)

The continuous delivery of modern software requires the execution of many automated pipeline jobs. These jobs ensure the frequent release of new software versions while detecting code problems at an early stage. For TELUS, our industrial partner in the telecommunications field, reliable job execution is crucial to minimize wasted time and streamline Continuous Deployment (CD). In this context, flaky job failures are one of the main issues hindering CD. Prior studies proposed techniques based on machine learning to automate the detection of flaky jobs. While valuable, these solutions are insufficient to address the waste associated with the diagnosis of flaky failures, which remain largely unexplored due to the wide range of underlying causes. This study examines 4,511 flaky job failures at TELUS to identify the different categories of flaky failures that we prioritize based on Recency, Frequency, and Monetary (RFM) measures. We identified 46 flaky failure categories that we analyzed using clustering and RFM measures to determine 14 priority categories for future automated diagnosis and repair research. Our findings also provide valuable insights into the evolution and impact of these categories. The identification and prioritization of flaky failure categories using RFM analysis introduce a novel approach that can be used in other contexts.