# Hierarchical Autoencoder-based Lossy Compression for Large-scale High-resolution Scientific Data

arXiv: https://arxiv.org/abs/2307.04216 · HTML: https://mlsanity.com/paper/2307.04216/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 70 references that are in the map.
- First posted on arXiv 9 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Density Modeling of Images using a Generalized Normalization Transformation](https://mlsanity.com/paper/1511.06281/) (2015)
- [ACORN: Adaptive Coordinate Networks for Neural Scene Representation](https://mlsanity.com/paper/2105.02788/) (2021)
- [Computationally Efficient Neural Image Compression](https://mlsanity.com/paper/1912.08771/) (2019)
- [Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Models](https://mlsanity.com/paper/2204.03497/) (2022)
- [Efficient Nonlinear Transforms for Lossy Image Compression](https://mlsanity.com/paper/1802.00847/) (2018)
- [NeuralVDB: High-resolution Sparse Volume Representation using Hierarchical Neural Networks](https://mlsanity.com/paper/2208.04448/) (2022)
- [Exploring Autoencoder-based Error-bounded Compression for Scientific Data](https://mlsanity.com/paper/2105.11730/) (2021)
- [SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors](https://mlsanity.com/paper/2101.03201/) (2021)
- [DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression](https://mlsanity.com/paper/1901.09124/) (2019)

## Built on by (most cited first, 1 of 1)

- [ANTIC: Adaptive Neural Temporal In-situ Compressor](https://mlsanity.com/paper/2604.09543/) (2026)

## Abstract (arXiv)

Lossy compression has become an important technique to reduce data size in many domains. This type of compression is especially valuable for large-scale scientific data, whose size ranges up to several petabytes. Although Autoencoder-based models have been successfully leveraged to compress images and videos, such neural networks have not widely gained attention in the scientific data domain. Our work presents a neural network that not only significantly compresses large-scale scientific data, but also maintains high reconstruction quality. The proposed model is tested with scientific benchmark data available publicly and applied to a large-scale high-resolution climate modeling data set. Our model achieves a compression ratio of 140 on several benchmark data sets without compromising the reconstruction quality. 2D simulation data from the High-Resolution Community Earth System Model (CESM) Version 1.3 over 500 years are also being compressed with a compression ratio of 200 while the reconstruction error is negligible for scientific analysis.