# Carbon Filter: Real-time Alert Triage Using Large Scale Clustering and Fast Search

arXiv: https://arxiv.org/abs/2405.04691 · HTML: https://mlsanity.com/paper/2405.04691/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 43 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 5 of 5)

- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [HOLMES: Real-time APT Detection through Correlation of Suspicious Information Flows](https://mlsanity.com/paper/1810.01594/) (2018)
- [Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance](https://mlsanity.com/paper/2308.05034/) (2023)
- [Lempel-Ziv Jaccard Distance, an Effective Alternative to Ssdeep and Sdhash](https://mlsanity.com/paper/1708.03346/) (2017)
- [FISHDBC: Flexible, Incremental, Scalable, Hierarchical Density-Based Clustering for Arbitrary Data and Distance](https://mlsanity.com/paper/1910.07283/) (2019)

## Built on by (most cited first, 1 of 1)

- [Positive-Unlabeled Learning for Agent Safety False Alarm Auditing](https://mlsanity.com/paper/2610.02925/) (2026)

## Abstract (arXiv)

"Alert fatigue" is one of the biggest challenges faced by the Security Operations Center (SOC) today, with analysts spending more than half of their time reviewing false alerts. Endpoint detection products raise alerts by pattern matching on event telemetry against behavioral rules that describe potentially malicious behavior, but can suffer from high false positives that distract from actual attacks. While alert triage techniques based on data provenance may show promise, these techniques can take over a minute to inspect a single alert, while EDR customers may face tens of millions of alerts per day; the current reality is that these approaches aren't nearly scalable enough for production environments. We present Carbon Filter, a statistical learning based system that dramatically reduces the number of alerts analysts need to manually review. Our approach is based on the observation that false alert triggers can be efficiently identified and separated from suspicious behaviors by examining the process initiation context (e.g., the command line) that launched the responsible process. Through the use of fast-search algorithms for training and inference, our approach scales to millions of alerts per day. Through batching queries to the model, we observe a theoretical maximum throughput of 20 million alerts per hour. Based on the analysis of tens of million alerts from customer deployments, our solution resulted in a 6-fold improvement in the Signal-to-Noise ratio without compromising on alert triage performance.