# Variable Rate Neural Compression for Sparse Detector Data

arXiv: https://arxiv.org/abs/2411.11942 · HTML: https://mlsanity.com/paper/2411.11942/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotations](https://mlsanity.com/paper/2002.12687/) (2020)
- [LHC analysis-specific datasets with Generative Adversarial Networks](https://mlsanity.com/paper/1901.05282/) (2019)
- [Simulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN)](https://mlsanity.com/paper/2001.11103/) (2020)
- [A reconfigurable neural network ASIC for detector front-end data compression at the HL-LHC](https://mlsanity.com/paper/2105.01683/) (2021)
- [cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data](https://mlsanity.com/paper/2007.09625/) (2020)
- [Keypoint-Aligned Embeddings for Image Retrieval and Re-identification](https://mlsanity.com/paper/2008.11368/) (2020)
- [MGARD+: Optimizing Multilevel Methods for Error-bounded Scientific Data Reduction](https://mlsanity.com/paper/2010.05872/) (2020)
- [Efficient Data Compression for 3D Sparse TPC via Bicephalous Convolutional Autoencoder](https://mlsanity.com/paper/2111.05423/) (2021)
- [3D Hand Pose and Shape Estimation from RGB Images for Keypoint-Based Hand Gesture Recognition](https://mlsanity.com/paper/2109.13879/) (2021)
- [Optimizing Error-Bounded Lossy Compression for Scientific Data on GPUs](https://mlsanity.com/paper/2105.12912/) (2021)
- [Machine-Learning Compression for Particle Physics Discoveries](https://mlsanity.com/paper/2210.11489/) (2022)
- [Fast 2D Bicephalous Convolutional Autoencoder for Compressing 3D Time Projection Chamber Data](https://mlsanity.com/paper/2310.15026/) (2023)
- [Differentiable Earth Mover's Distance for Data Compression at the High-Luminosity LHC](https://mlsanity.com/paper/2306.04712/) (2023)
- [cuSZ-i: High-Ratio Scientific Lossy Compression on GPUs with Optimized Multi-Level Interpolation](https://mlsanity.com/paper/2312.05492/) (2023)

## Abstract (arXiv)

High-energy large-scale particle colliders generate data at extraordinary rates. Developing real-time high-throughput data compression algorithms to reduce data volume and meet the bandwidth requirement for storage has become increasingly critical. Deep learning is a promising technology that can address this challenging topic. At the newly constructed sPHENIX experiment at the Relativistic Heavy Ion Collider, a Time Projection Chamber (TPC) serves as the main tracking detector, which records three-dimensional particle trajectories in a volume of a gas-filled cylinder. In terms of occupancy, the resulting data flow can be very sparse reaching 10-3 for proton-proton collisions. Such sparsity presents a challenge to conventional learning-free lossy compression algorithms, such as SZ, ZFP, and MGARD. In contrast, emerging deep learning-based models, particularly those utilizing convolutional neural networks for compression, have outperformed these conventional methods in terms of compression ratios and reconstruction accuracy. However, research on the efficacy of these deep learning models in handling sparse datasets, like those produced in particle colliders, remains limited. Furthermore, most deep learning models do not adapt their processing speeds to data sparsity, which affects efficiency. To address this issue, we propose a novel approach for TPC data compression via key-point identification facilitated by sparse convolution. Our proposed algorithm, BCAE-VS, achieves a 75% improvement in reconstruction accuracy with a 10% increase in compression ratio over the previous state-of-the-art model. Additionally, BCAE-VS manages to achieve these results with a model size over two orders of magnitude smaller. Lastly, we have experimentally verified that as sparsity increases, so does the model's throughput.