# DeCaf: Diagnosing and Triaging Performance Issues in Large-Scale Cloud Services

arXiv: https://arxiv.org/abs/1910.05339 · HTML: https://mlsanity.com/paper/1910.05339/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 16 of 16)

- [Exploring LLM-based Agents for Root Cause Analysis](https://mlsanity.com/paper/2403.04123/) (2024)
- [Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models](https://mlsanity.com/paper/2301.03797/) (2023)
- [Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4](https://mlsanity.com/paper/2401.13810/) (2024)
- [X-lifecycle Learning for Cloud Incident Management using LLMs](https://mlsanity.com/paper/2404.03662/) (2024)
- [Building AI Agents for Autonomous Clouds: Challenges and Design Principles](https://mlsanity.com/paper/2407.12165/) (2024)
- [AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges](https://mlsanity.com/paper/2304.04661/) (2023)
- [AutoTSG: Learning and Synthesis for Incident Troubleshooting](https://mlsanity.com/paper/2205.13457/) (2022)
- [A Comprehensive Survey on Root Cause Analysis in (Micro) Services: Methodologies, Challenges, and Trends](https://mlsanity.com/paper/2408.00803/) (2024)
- [Towards a consistent interpretation of AIOps models](https://mlsanity.com/paper/2202.02298/) (2022)
- [Mining Root Cause Knowledge from Cloud Service Incident Investigations for AIOps](https://mlsanity.com/paper/2204.11598/) (2022)
- [SoftNER: Mining Knowledge Graphs From Cloud Incidents](https://mlsanity.com/paper/2101.05961/) (2021)
- [Simplifying Root Cause Analysis in Kubernetes with StateGraph and LLM](https://mlsanity.com/paper/2506.02490/) (2025)
- [Dependency Aware Incident Linking in Large Cloud Systems](https://mlsanity.com/paper/2403.18639/) (2024)
- [DeLag: Using Multi-Objective Optimization to Enhance the Detection of Latency Degradation Patterns in Service-based Systems](https://mlsanity.com/paper/2110.11155/) (2021)
- [KGroot: Enhancing Root Cause Analysis through Knowledge Graphs and Graph Convolutional Neural Networks](https://mlsanity.com/paper/2402.13264/) (2024)
- [RiskLoc: Localization of Multi-dimensional Root Causes by Weighted Risk](https://mlsanity.com/paper/2205.10004/) (2022)

## Abstract (arXiv)

Large scale cloud services use Key Performance Indicators (KPIs) for tracking and monitoring performance. They usually have Service Level Objectives (SLOs) baked into the customer agreements which are tied to these KPIs. Dependency failures, code bugs, infrastructure failures, and other problems can cause performance regressions. It is critical to minimize the time and manual effort in diagnosing and triaging such issues to reduce customer impact. Large volume of logs and mixed type of attributes (categorical, continuous) in the logs makes diagnosis of regressions non-trivial. In this paper, we present the design, implementation and experience from building and deploying DeCaf, a system for automated diagnosis and triaging of KPI issues using service logs. It uses machine learning along with pattern mining to help service owners automatically root cause and triage performance issues. We present the learnings and results from case studies on two large scale cloud services in Microsoft where DeCaf successfully diagnosed 10 known and 31 unknown issues. DeCaf also automatically triages the identified issues by leveraging historical data. Our key insights are that for any such diagnosis tool to be effective in practice, it should a) scale to large volumes of service logs and attributes, b) support different types of KPIs and ranking functions, c) be integrated into the DevOps processes.