# High-Ratio Compression for Machine-Generated Data

arXiv: https://arxiv.org/abs/2311.13947 · HTML: https://mlsanity.com/paper/2311.13947/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2023.

## Built on by (most cited first, 3 of 3)

- [How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction](https://mlsanity.com/paper/2509.01914/) (2025)
- [SINDI: An Efficient Index for Sparse Vector Approximate Maximum Inner Product Search](https://mlsanity.com/paper/2509.08395/) (2025)
- [Morphing-based Compression for Data-centric ML Pipelines](https://mlsanity.com/paper/2504.11067/) (2025)

## Abstract (arXiv)

Machine-generated data is rapidly growing and poses challenges for data-intensive systems, especially as the growth of data outpaces the growth of storage space. To cope with the storage issue, compression plays a critical role in storage engines, particularly for data-intensive applications, where high compression ratios and efficient random access are essential. However, existing compression techniques tend to focus on general-purpose and data block approaches, but overlook the inherent structure of machine-generated data and hence result in low-compression ratios or limited lookup efficiency. To address these limitations, we introduce the Pattern-Based Compression (PBC) algorithm, which specifically targets patterns in machine-generated data to achieve Pareto-optimality in most cases. Unlike traditional data block-based methods, PBC compresses data on a per-record basis, facilitating rapid random access. Our experimental evaluation demonstrates that PBC, on average, achieves a compression ratio twice as high as state-of-the-art techniques while maintaining competitive compression and decompression speeds.We also integrate PBC to a production database system and achieve improvement on both comparison ratio and throughput.