# Real-time Workload Pattern Analysis for Large-scale Cloud Databases

arXiv: https://arxiv.org/abs/2307.02626 · HTML: https://mlsanity.com/paper/2307.02626/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 63 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Neo: A Learned Query Optimizer](https://mlsanity.com/paper/1904.03711/) (2019)
- [Bao: Learning to Steer Query Optimizers](https://mlsanity.com/paper/2004.03814/) (2020)
- [LargeEA: Aligning Entities for Large-scale Knowledge Graphs](https://mlsanity.com/paper/2108.05211/) (2021)
- [ClusterEA: Scalable Entity Alignment with Stochastic Training and Normalized Mini-batch Similarities](https://mlsanity.com/paper/2205.10312/) (2022)
- [Unsupervised Entity Alignment for Temporal Knowledge Graphs](https://mlsanity.com/paper/2302.00796/) (2023)
- [Query2Vec: An Evaluation of NLP Techniques for Generalized Workload Analytics](https://mlsanity.com/paper/1801.05613/) (2018)
- [Data Generation for Testing and Grading SQL Queries](https://mlsanity.com/paper/1411.6704/) (2014)
- [Comprehensive and Efficient Workload Compression](https://mlsanity.com/paper/2011.05549/) (2020)
- [SEA: A Scalable Entity Alignment System](https://mlsanity.com/paper/2304.07065/) (2023)
- [WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases](https://mlsanity.com/paper/1601.08221/) (2016)
- [Facilitating SQL Query Composition and Analysis](https://mlsanity.com/paper/2002.09091/) (2020)
- [Database Workload Characterization with Query Plan Encoders](https://mlsanity.com/paper/2105.12287/) (2021)
- [KEA: Tuning an Exabyte-Scale Data Infrastructure](https://mlsanity.com/paper/2106.11445/) (2021)
- [Guided Automated Learning for query workload re-Optimization](https://mlsanity.com/paper/1901.02049/) (2019)

## Built on by (most cited first, 4 of 4)

- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads](https://mlsanity.com/paper/2507.06192/) (2025)
- [Moon: A Modality Conversion-based Efficient Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2510.01970/) (2025)
- [LearnedWMP: Workload Memory Prediction Using Distribution of Query Templates](https://mlsanity.com/paper/2401.12103/) (2024)

## Abstract (arXiv)

Hosting database services on cloud systems has become a common practice. This has led to the increasing volume of database workloads, which provides the opportunity for pattern analysis. Discovering workload patterns from a business logic perspective is conducive to better understanding the trends and characteristics of the database system. However, existing workload pattern discovery systems are not suitable for large-scale cloud databases which are commonly employed by the industry. This is because the workload patterns of large-scale cloud databases are generally far more complicated than those of ordinary databases. In this paper, we propose Alibaba Workload Miner (AWM), a real-time system for discovering workload patterns in complicated large-scale workloads. AWM encodes and discovers the SQL query patterns logged from user requests and optimizes the querying processing based on the discovered patterns. First, Data Collection & Preprocessing Module collects streaming query logs and encodes them into high-dimensional feature embeddings with rich semantic contexts and execution features. Next, Online Workload Mining Module separates encoded queries by business groups and discovers the workload patterns for each group. Meanwhile, Offline Training Module collects labels and trains the classification model using the labels. Finally, Pattern-based Optimizing Module optimizes query processing in cloud databases by exploiting discovered patterns. Extensive experimental results on one synthetic dataset and two real-life datasets (extracted from Alibaba Cloud databases) show that AWM enhances the accuracy of pattern discovery by 66% and reduce the latency of online inference by 22%, compared with the state-of-the-arts.