# Smart Compaction: Predicting Compaction Utility from Lakehouse Table Metadata

arXiv: https://arxiv.org/abs/2608.08639 · HTML: https://mlsanity.com/paper/2608.08639/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Builds on (most cited first, 7 of 7)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Bao: Learning to Steer Query Optimizers](https://mlsanity.com/paper/2004.03814/) (2020)
- [Qd-tree: Learning Data Layouts for Big Data Analytics](https://mlsanity.com/paper/2004.10898/) (2020)
- [Learning to Optimize LSM-trees: Towards A Reinforcement Learning based Key-Value Store for Dynamic Workloads](https://mlsanity.com/paper/2308.07013/) (2023)
- [Constructing and Analyzing the LSM Compaction Design Space (Updated Version)](https://mlsanity.com/paper/2202.04522/) (2022)
- [LST-Bench: Benchmarking Log-Structured Tables in the Cloud](https://mlsanity.com/paper/2305.01120/) (2023)
- [AutoComp: Automated Data Compaction for Log-Structured Tables in Data Lakes](https://mlsanity.com/paper/2504.04186/) (2025)

## Abstract (arXiv)

Open lakehouse table formats accumulate small data files over time, which degrades query performance. Deciding when compaction is worthwhile remains threshold-driven, but which metadata features actually determine compaction utility is not well understood. We present an open simulation framework that generates 2,376 Apache Iceberg tables spanning three orders of magnitude in file size, extracts 17 metadata features from manifest files without reading data, and trains XGBoost to predict the continuous file-reduction ratio (R2 = 0.998, RMSE= 0.013). The binary compaction decision turns out to be trivially separable by a single partition-level threshold max_files_per_partition> 4, requiring no learned model. Cross-schema validation on 96 TPC-H tables confirms generalisation without retraining (R2 = 0.976). A query benchmark reveals that compaction benefits metadata-heavy queries but can slow full-scan aggregations by reducing task parallelism. All code and data are publicly available.