# Attention Based Machine Learning Methods for Data Reduction with Guaranteed Error Bounds

arXiv: https://arxiv.org/abs/2409.05357 · HTML: https://mlsanity.com/paper/2409.05357/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Weak SINDy For Partial Differential Equations](https://mlsanity.com/paper/2007.02848/) (2020)
- [TTHRESH: Tensor Compression for Multidimensional Visual Data](https://mlsanity.com/paper/1806.05952/) (2018)
- [SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors](https://mlsanity.com/paper/2111.02925/) (2021)
- [Compression with Bayesian Implicit Neural Representations](https://mlsanity.com/paper/2305.19185/) (2023)
- [Exploring Autoencoder-based Error-bounded Compression for Scientific Data](https://mlsanity.com/paper/2105.11730/) (2021)
- [MGARD: A multigrid framework for high-performance, error-controlled data compression and refactoring](https://mlsanity.com/paper/2401.05994/) (2024)
- [QuadConv: Quadrature-Based Convolutions with Applications to Non-Uniform PDE Data Compression](https://mlsanity.com/paper/2211.05151/) (2022)
- [Streaming Compression of Scientific Data via weak-SINDy](https://mlsanity.com/paper/2308.14962/) (2023)
- [SRN-SZ: Deep Leaning-Based Scientific Error-bounded Lossy Compression with Super-resolution Neural Networks](https://mlsanity.com/paper/2309.04037/) (2023)
- [Machine Learning Techniques for Data Reduction of CFD Applications](https://mlsanity.com/paper/2404.18063/) (2024)

## Built on by (most cited first, 5 of 5)

- [Foundation Model for Lossy Compression of Spatiotemporal Scientific Data](https://mlsanity.com/paper/2412.17184/) (2024)
- [Generative Latent Diffusion for Efficient Spatiotemporal Data Reduction](https://mlsanity.com/paper/2507.02129/) (2025)
- [Residual Modeling for High-Fidelity Learned Compression of Scientific Data](https://mlsanity.com/paper/2606.05389/) (2026)
- [Can Deep Neural Networks Improve Compression of Very Large Scientific Data?](https://mlsanity.com/paper/2606.14353/) (2026)
- [Neural Residual Modeling for Scientific Data Compression under Guaranteed Error Bounds](https://mlsanity.com/paper/2609.23185/) (2026)

## Abstract (arXiv)

Scientific applications in fields such as high energy physics, computational fluid dynamics, and climate science generate vast amounts of data at high velocities. This exponential growth in data production is surpassing the advancements in computing power, network capabilities, and storage capacities. To address this challenge, data compression or reduction techniques are crucial. These scientific datasets have underlying data structures that consist of structured and block structured multidimensional meshes where each grid point corresponds to a tensor. It is important that data reduction techniques leverage strong spatial and temporal correlations that are ubiquitous in these applications. Additionally, applications such as CFD, process tensors comprising hundred plus species and their attributes at each grid point. Reduction techniques should be able to leverage interrelationships between the elements in each tensor. In this paper, we propose an attention-based hierarchical compression method utilizing a block-wise compression setup. We introduce an attention-based hyper-block autoencoder to capture inter-block correlations, followed by a block-wise encoder to capture block-specific information. A PCA-based post-processing step is employed to guarantee error bounds for each data block. Our method effectively captures both spatiotemporal and inter-variable correlations within and between data blocks. Compared to the state-of-the-art SZ3, our method achieves up to 8 times higher compression ratio on the multi-variable S3D dataset. When evaluated on single-variable setups using the E3SM and XGC datasets, our method still achieves up to 3 times and 2 times higher compression ratio, respectively.