# A Fast Transformer-based General-Purpose Lossless Compressor

arXiv: https://arxiv.org/abs/2203.16114 · HTML: https://mlsanity.com/paper/2203.16114/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Practical Lossless Compression with Latent Variables using Bits Back Coding](https://mlsanity.com/paper/1901.04866/) (2019)
- [Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables](https://mlsanity.com/paper/1905.06845/) (2019)
- [IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression](https://mlsanity.com/paper/2006.12459/) (2020)
- [Sub-Linear Memory: How to Make Performers SLiM](https://mlsanity.com/paper/2012.11346/) (2020)
- [DZip: improved general-purpose lossless compression based on novel neural network modeling](https://mlsanity.com/paper/1911.03572/) (2019)
- [Not all parameters are born equal: Attention is mostly what you need](https://mlsanity.com/paper/2010.11859/) (2020)

## Built on by (most cited first, 26 of 26)

- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data](https://mlsanity.com/paper/2410.05078/) (2024)
- [NTIRE 2025 Challenge on Image Super-Resolution (x4): Methods and Results](https://mlsanity.com/paper/2504.14582/) (2025)
- [GeneFormer: Learned Gene Compression using Transformer-based Context Modeling](https://mlsanity.com/paper/2212.08379/) (2022)
- [Approximating Human-Like Few-shot Learning with GPT-based Compression](https://mlsanity.com/paper/2308.06942/) (2023)
- [Towards Optimal Learning of Language Models](https://mlsanity.com/paper/2402.17759/) (2024)
- [On the Compressibility of Quantized Large Language Models](https://mlsanity.com/paper/2403.01384/) (2024)
- [GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model](https://mlsanity.com/paper/2408.15868/) (2024)
- [Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction](https://mlsanity.com/paper/2505.06297/) (2025)
- [Debiasing Machine Unlearning with Counterfactual Examples](https://mlsanity.com/paper/2404.15760/) (2024)
- [NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression](https://mlsanity.com/paper/2409.05785/) (2024)
- [WISE: A Framework for Gigapixel Whole-Slide-Image Lossless Compression](https://mlsanity.com/paper/2503.18074/) (2025)
- [Ranking LLMs by compression](https://mlsanity.com/paper/2406.14171/) (2024)
- [An Enhanced Text Compression Approach Using Transformer-based Language Models](https://mlsanity.com/paper/2412.15250/) (2024)
- [An Efficient Compression of Deep Neural Network Checkpoints Based on Prediction and Context Modeling](https://mlsanity.com/paper/2506.12000/) (2025)
- [PMKLC: Parallel Multi-Knowledge Learning-based Lossless Compression for Large-Scale Genomics Database](https://mlsanity.com/paper/2507.12805/) (2025)
- [EcoSense: Energy-Efficient Intelligent Sensing for In-Shore Ship Detection through Edge-Cloud Collaboration](https://mlsanity.com/paper/2403.14027/) (2024)
- [In-Context Source and Channel Coding](https://mlsanity.com/paper/2601.10267/) (2026)
- [AgentGC: Evolutionary Learning-based Lossless Compression for Genomics Data with LLM-driven Multiple Agent](https://mlsanity.com/paper/2601.13559/) (2026)
- [RW-NSGCN: A Robust Approach to Structural Attacks via Negative Sampling](https://mlsanity.com/paper/2408.06665/) (2024)
- [Easz: An Agile Transformer-based Image Compression Framework for Resource-constrained IoTs](https://mlsanity.com/paper/2505.01742/) (2025)
- [Lossless Compression: A New Benchmark for Time Series Model Evaluation](https://mlsanity.com/paper/2509.21002/) (2025)
- [A Model-Driven Lossless Compression Algorithm Resistant to Mismatch](https://mlsanity.com/paper/2601.17684/) (2026)
- [Efficient Learned Data Compression via Dual-Stream Feature Decoupling](https://mlsanity.com/paper/2604.07239/) (2026)
- [Lossless Compression via Chained Lightweight Neural Predictors with Information Inheritance](https://mlsanity.com/paper/2604.15472/) (2026)
- [TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding](https://mlsanity.com/paper/2606.08184/) (2026)

## Abstract (arXiv)

Deep-learning-based compressor has received interests recently due to much improved compression ratio. However, modern approaches suffer from long execution time. To ease this problem, this paper targets on cutting down the execution time of deep-learning-based compressors. Building history-dependencies sequentially (e.g., recurrent neural networks) is responsible for long inference latency. Instead, we introduce transformer into deep learning compressors to build history-dependencies in parallel. However, existing transformer is too heavy in computation and incompatible to compression tasks. This paper proposes a fast general-purpose lossless compressor, TRACE, by designing a compression-friendly structure based on a single-layer transformer. We first design a new metric to advise the selection part of compression model structures. Byte-grouping and Shared-ffn schemes are further proposed to fully utilize the capacity of the single-layer transformer. These features allow TRACE to achieve competitive compression ratio and a much faster speed. In addition, we further accelerate the compression procedure by designing a controller to reduce the parameter updating overhead. Experiments show that TRACE achieves an overall ~3x speedup while keeps a comparable compression ratio to the state-of-the-art compressors. The source code for TRACE and links to the datasets are available at https://github.com/mynotwo/A-Fast-Transformer-based-General-Purpose-LosslessCompressor.