# RankShift: In-Database Detection and Explanation of Categorical Shifts

arXiv: https://arxiv.org/abs/2608.28922 · HTML: https://mlsanity.com/paper/2608.28922/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 9 of 9)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [LogBERT: Log Anomaly Detection via BERT](https://mlsanity.com/paper/2103.04475/) (2021)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [MIDAS: Microcluster-Based Detector of Anomalies in Edge Streams](https://mlsanity.com/paper/1911.04464/) (2019)
- [DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications](https://mlsanity.com/paper/2109.11495/) (2021)
- [MSTREAM: Fast Anomaly Detection in Multi-Aspect Streams](https://mlsanity.com/paper/2009.08451/) (2020)
- [MacroBase: Prioritizing Attention in Fast Data](https://mlsanity.com/paper/1603.00567/) (2016)
- [ACE -- An Anomaly Contribution Explainer for Cyber-Security Applications](https://mlsanity.com/paper/1912.00314/) (2019)

## Abstract (arXiv)

A login service can receive its usual number of failed sign-ins while one source grows from 2% to 30% of them. The same pattern appears in system logs when a rare event template becomes common while the message rate stays stable. These events change which categories are active without changing how many events occur. RankShift detects such changes inside the analytical database that stores the data. It compares each window's category shares with a benign reference using a Pearson score whose terms identify the categories responsible for the change. The same query returns the score, calibrated alert, and largest increasing contributions. We evaluate RankShift on HDFS, BGL, and Thunderbird. It matches the count-vector autoencoder within 0.001 AUROC on HDFS (0.999 versus 1.000) and leads on Thunderbird (0.983 versus 0.949). In a controlled fixed-volume experiment, RankShift detects rare-category shifts that are invisible to event-count monitoring, reaching 0.787 AUROC compared with 0.771 for the autoencoder. Across all three corpora, observed false-alarm rates track the requested operating levels. RankShift requires no model training or inference service, and the autoencoders deployed state is 137x larger.