# Dynamic Quality Metric Oriented Error-bounded Lossy Compression for Scientific Datasets

arXiv: https://arxiv.org/abs/2310.14133 · HTML: https://mlsanity.com/paper/2310.14133/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2023.

## Built on by (most cited first, 2 of 2)

- [GWLZ: A Group-wise Learning-based Lossy Compression Framework for Scientific Data](https://mlsanity.com/paper/2404.13470/) (2024)
- [NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression](https://mlsanity.com/paper/2409.05785/) (2024)

## Abstract (arXiv)

With the ever-increasing execution scale of high performance computing (HPC) applications, vast amounts of data are being produced by scientific research every day. Error-bounded lossy compression has been considered a very promising solution to address the big-data issue for scientific applications because it can significantly reduce the data volume with low time cost meanwhile allowing users to control the compression errors with a specified error bound. The existing error-bounded lossy compressors, however, are all developed based on inflexible designs or compression pipelines, which cannot adapt to diverse compression quality requirements/metrics favored by different application users. In this paper, we propose a novel dynamic quality metric oriented error-bounded lossy compression framework, namely QoZ. The detailed contribution is three-fold. (1) We design a novel highly-parameterized multi-level interpolation-based data predictor, which can significantly improve the overall compression quality with the same compressed size. (2) We design the error-bounded lossy compression framework QoZ based on the adaptive predictor, which can auto-tune the critical parameters and optimize the compression result according to user-specified quality metrics during online compression. (3) We evaluate QoZ carefully by comparing its compression quality with multiple state-of-the-arts on various real-world scientific application datasets. Experiments show that, compared with the second-best lossy compressor, QoZ can achieve up to 70% compression ratio improvement under the same error bound, up to 150% compression ratio improvement under the same PSNR, or up to 270% compression ratio improvement under the same SSIM.