# An Enhanced Text Compression Approach Using Transformer-based Language Models

arXiv: https://arxiv.org/abs/2412.15250 · HTML: https://mlsanity.com/paper/2412.15250/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Tokenization Is More Than Compression](https://mlsanity.com/paper/2402.18376/) (2024)
- [LLMZip: Lossless Text Compression using Large Language Models](https://mlsanity.com/paper/2306.04050/) (2023)
- [DeepZip: Lossless Data Compression using Recurrent Neural Networks](https://mlsanity.com/paper/1811.08162/) (2018)
- [A Fast Transformer-based General-Purpose Lossless Compressor](https://mlsanity.com/paper/2203.16114/) (2022)
- [Lossless Acceleration for Seq2seq Generation with Aggressive Decoding](https://mlsanity.com/paper/2205.10350/) (2022)
- [Text Compression-aided Transformer Encoding](https://mlsanity.com/paper/2102.05951/) (2021)
- [Explicit Sentence Compression for Neural Machine Translation](https://mlsanity.com/paper/1912.11980/) (2019)
- [TexShape: Information Theoretic Sentence Embedding for Language Models](https://mlsanity.com/paper/2402.05132/) (2024)
- [Approximating Human-Like Few-shot Learning with GPT-based Compression](https://mlsanity.com/paper/2308.06942/) (2023)
- [Crossword: A Semantic Approach to Data Compression via Masking](https://mlsanity.com/paper/2304.01106/) (2023)
- [Graph Neural Network and NER-Based Text Summarization](https://mlsanity.com/paper/2402.05126/) (2024)

## Built on by (most cited first, 2 of 2)

- [M+: Extending MemoryLLM with Scalable Long-Term Memory](https://mlsanity.com/paper/2502.00592/) (2025)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)

## Abstract (arXiv)

Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy. The integration of lossless compression techniques with transformer-based text decompression has received negligible attention, despite the increasing volume of English text data in communication. The primary barrier in advancing text compression and restoration involves optimizing transformer-based approaches with efficient pre-processing and integrating lossless compression algorithms, that remained unresolved in the prior attempts. Here, we propose a transformer-based method named RejuvenateForme for text decompression, addressing prior issues by harnessing a new pre-processing technique and a lossless compression method. Our meticulous pre-processing technique incorporating the Lempel-Ziv-Welch algorithm achieves compression ratios of 12.57, 13.38, and 11.42 on the BookCorpus, EN-DE, and EN-FR corpora, thus showing state-of-the-art compression ratios compared to other deep learning and traditional approaches. Furthermore, the RejuvenateForme achieves a BLEU score of 27.31, 25.78, and 50.45 on the EN-DE, EN-FR, and BookCorpus corpora, showcasing its comprehensive efficacy. In contrast, the pre-trained T5-Small exhibits better performance over prior state-of-the-art models.