# Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

arXiv: https://arxiv.org/abs/1910.00409 · HTML: https://mlsanity.com/paper/1910.00409/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2019.

## Built on by (most cited first, 14 of 14)

- [AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges](https://mlsanity.com/paper/2304.04661/) (2023)
- [UniLog: Deploy One Model and Specialize it for All Log Analysis Tasks](https://mlsanity.com/paper/2112.03159/) (2021)
- [Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study](https://mlsanity.com/paper/2409.04834/) (2024)
- [A Comprehensive Survey of Logging in Software: From Logging Statements Automation to Log Mining and Analysis](https://mlsanity.com/paper/2110.12489/) (2021)
- [Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach](https://mlsanity.com/paper/2403.06485/) (2024)
- [Graph-based Incident Aggregation for Large-Scale Online Service Systems](https://mlsanity.com/paper/2108.12179/) (2021)
- [Log-based Anomaly Detection based on EVT Theory with feedback](https://mlsanity.com/paper/2306.05032/) (2023)
- [Deep Learning-based Intrusion Detection Systems: A Survey](https://mlsanity.com/paper/2504.07839/) (2025)
- [Linnaeus: A highly reusable and adaptable ML based log classification pipeline](https://mlsanity.com/paper/2103.06927/) (2021)
- [Try with Simpler -- An Evaluation of Improved Principal Component Analysis in Log-based Anomaly Detection](https://mlsanity.com/paper/2308.12612/) (2023)
- [Task-Aware Reduction for Scalable LLM-Database Systems](https://mlsanity.com/paper/2510.11813/) (2025)
- [LogSieve: Task-Aware CI Log Reduction for Sustainable LLM-Based Analysis](https://mlsanity.com/paper/2601.20148/) (2026)
- [CLAD: Efficient Log Anomaly Detection Directly on Compressed Representations](https://mlsanity.com/paper/2604.13024/) (2026)
- [FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection](https://mlsanity.com/paper/2605.22779/) (2026)

## Abstract (arXiv)

System logs record detailed runtime information of software systems and are used as the main data source for many tasks around software engineering. As modern software systems are evolving into large scale and complex structures, logs have become one type of fast-growing big data in industry. In particular, such logs often need to be stored for a long time in practice (e.g., a year), in order to analyze recurrent problems or track security issues. However, archiving logs consumes a large amount of storage space and computing resources, which in turn incurs high operational cost. Data compression is essential to reduce the cost of log storage. Traditional compression tools (e.g., gzip) work well for general texts, but are not tailed for system logs. In this paper, we propose a novel and effective log compression method, namely logzip. Logzip is capable of extracting hidden structures from raw logs via fast iterative clustering and further generating coherent intermediate representations that allow for more effective compression. We evaluate logzip on five large log datasets of different system types, with a total of 63.6 GB in size. The results show that logzip can save about half of the storage space on average over traditional compression tools. Meanwhile, the design of logzip is highly parallel and only incurs negligible overhead. In addition, we share our industrial experience of applying logzip to Huawei's real products.