# Can Deep Neural Networks Improve Compression of Very Large Scientific Data?

arXiv: https://arxiv.org/abs/2606.14353 · HTML: https://mlsanity.com/paper/2606.14353/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 42 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 5 of 5)

- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [Exploring Autoencoder-based Error-bounded Compression for Scientific Data](https://mlsanity.com/paper/2105.11730/) (2021)
- [High-performance Effective Scientific Error-bounded Lossy Compression with Auto-tuned Multi-component Interpolation](https://mlsanity.com/paper/2311.12133/) (2023)
- [Attention Based Machine Learning Methods for Data Reduction with Guaranteed Error Bounds](https://mlsanity.com/paper/2409.05357/) (2024)
- [Aardvark weather: end-to-end data-driven weather forecasting](https://mlsanity.com/paper/2404.00411/) (2024)

## Abstract (arXiv)

Error-bounded lossy compression is a fundamental technique for managing the rapidly growing volumes of scientific data produced by modern simulations and observational instruments. Most state-of-the-art-compressors follow a prediction-residual paradigm, where compression effectiveness depends on the quality of the predictor: more accurate predictions generate smaller residuals that are easier to compress. This observation raises a question: can modern machine learning models serve as superior predictors for scientific data compression? Answering this question directly is challenging because developing compression-specific ML predictors requires substantial resources. Instead, we leverage the climate domain where highly accurate pretrained weather forecasting foundation models already exist, making them an ideal testbed. We present a framework that integrates spatial and temporal deep learning models into a conventional error-bounded compression pipeline. The framework supports auto-regressive forecasting models and avoids error accumulation. Using ERA5 climate data as a representative large-scale scientific dataset, we evaluate three distinct ML predictors: a VAEformer-based codec (CRA5), a graph neural network forecaster (GraphCast), and a vision-transformer forecaster (Aurora), against the state-of-the-art compressor SZ3.1 under identical quantization and entropy-coding backends. Our evaluation over approximately 1.7 TB of data reveals a surprising result: although ML predictors generate more accurate predictions and can improve reconstruction quality by up to 91% while achieving up to 9.6x higher compression ratios for highly predictable variables, they do not improve overall dataset-level compression ratio. We show that prediction accuracy alone is insufficient: the spatial structure of the resulting residuals plays a decisive role in entropy coding efficiency.