# Efficient Detection of Intermittent Job Failures Using Few-Shot Learning

arXiv: https://arxiv.org/abs/2507.04173 · HTML: https://mlsanity.com/paper/2507.04173/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs](https://mlsanity.com/paper/2412.13337/) (2024)
- [A Theoretical Analysis of the Number of Shots in Few-Shot Learning](https://mlsanity.com/paper/1909.11722/) (2019)
- [What is the Vocabulary of Flaky Tests? An Extended Replication](https://mlsanity.com/paper/2103.12670/) (2021)
- [What Causes My Test Alarm? Automatic Cause Analysis for Test Alarms in System and Integration Testing](https://mlsanity.com/paper/1703.00768/) (2017)
- [Game Development Software Engineering Process Life Cycle: A Systematic Review](https://mlsanity.com/paper/1711.08527/) (2017)
- [Empirical Study of Restarted and Flaky Builds on Travis CI](https://mlsanity.com/paper/2003.11772/) (2020)
- [On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories](https://mlsanity.com/paper/2501.04976/) (2025)
- [Build Optimization: A Systematic Literature Review](https://mlsanity.com/paper/2501.11940/) (2025)

## Built on by (most cited first, 1 of 1)

- [Predicting Intermittent Job Failure Categories for Diagnosis Using Few-Shot Fine-Tuned Language Models](https://mlsanity.com/paper/2601.22264/) (2026)

## Abstract (arXiv)

One of the main challenges developers face in the use of continuous integration (CI) and deployment pipelines is the occurrence of intermittent job failures, which result from unexpected non-deterministic issues (e.g., flaky tests or infrastructure problems) rather than regular code-related errors such as bugs. Prior studies developed machine learning (ML) models trained on large datasets of job logs to classify job failures as either intermittent or regular. As an alternative to costly manual labeling of large datasets, the state-of-the-art (SOTA) approach leveraged a heuristic based on non-deterministic job reruns. However, this method mislabels intermittent job failures as regular in contexts where rerunning suspicious job failures is not an explicit policy, and therefore limits the SOTA's performance in practice. In fact, our manual analysis of 2,125 job failures from 5 industrial and 1 open-source projects reveals that, on average, 32% of intermittent job failures are mislabeled as regular. To address these limitations, this paper introduces a novel approach to intermittent job failure detection using few-shot learning (FSL). Specifically, we fine-tune a small language model using a few number of manually labeled log examples to generate rich embeddings, which are then used to train an ML classifier. Our FSL-based approach achieves 70-88% F1-score with only 12 shots in all projects, outperforming the SOTA, which proved ineffective (34-52% F1-score) in 4 projects. Overall, this study underlines the importance of data quality over quantity and provides a more efficient and practical framework for the detection of intermittent job failures in organizations.