# Context Cascade Compression: Exploring the Upper Limits of Text Compression

arXiv: https://arxiv.org/abs/2511.15244 · HTML: https://mlsanity.com/paper/2511.15244/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [DeepSeek-OCR: Contexts Optical Compression](https://mlsanity.com/paper/2510.18234/) (2025)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [Focus Anywhere for Fine-grained Multi-page Document Understanding](https://mlsanity.com/paper/2405.14295/) (2024)
- [Glyph: Scaling Context Windows via Visual-Text Compression](https://mlsanity.com/paper/2510.17800/) (2025)
- [In-Context Former: Lightning-fast Compressing Context for Large Language Model](https://mlsanity.com/paper/2406.13618/) (2024)

## Built on by (most cited first, 17 of 17)

- [DeepSeek-OCR 2: Visual Causal Flow](https://mlsanity.com/paper/2601.20552/) (2026)
- [Latent Context Compilation: Distilling Long Context into Compact Portable Memory](https://mlsanity.com/paper/2602.21221/) (2026)
- [VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?](https://mlsanity.com/paper/2512.15649/) (2025)
- [Context Distillation as Latent Memory Management](https://mlsanity.com/paper/2605.28889/) (2026)
- [From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition](https://mlsanity.com/paper/2512.14244/) (2025)
- [Global Context Compression with Interleaved Vision-Text Transformation](https://mlsanity.com/paper/2601.10378/) (2026)
- [Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inference](https://mlsanity.com/paper/2602.10021/) (2026)
- [Memory as Metabolism: A Design for Companion Knowledge Systems](https://mlsanity.com/paper/2604.12034/) (2026)
- [Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models](https://mlsanity.com/paper/2606.12203/) (2026)
- [CoreMem: Riemannian Retrieval and Fisher-Guided Distillation for Long-Term Memory in Dialogue Agents](https://mlsanity.com/paper/2606.18406/) (2026)
- [How Much Information Can a Vision Token Hold? A Scaling Law for Recognition Limits in VLMs](https://mlsanity.com/paper/2602.02539/) (2026)
- [How to Utilize Complementary Vision-Text Information for 2D Structure Understanding](https://mlsanity.com/paper/2603.16245/) (2026)
- [Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio](https://mlsanity.com/paper/2603.25926/) (2026)
- [GRC: Unifying Reasoning-Driven Generation, Retrieval and Compression](https://mlsanity.com/paper/2605.09100/) (2026)
- [Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models](https://mlsanity.com/paper/2609.15338/) (2026)
- [Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation](https://mlsanity.com/paper/2609.35333/) (2026)
- [Lossy Compressive Text Autoencoders](https://mlsanity.com/paper/2610.10738/) (2026)

## Abstract (arXiv)

Million-level token inputs in long-context tasks pose significant computational and memory challenges for Large Language Models (LLMs). Recently, DeepSeek-OCR conducted research into the feasibility of Contexts Optical Compression and achieved preliminary results. Inspired by this, we introduce Context Cascade Compression C3 to explore the upper limits of text compression. Our method cascades two LLMs of different sizes to handle the compression and decoding tasks. Specifically, a small LLM, acting as the first stage, performs text compression by condensing a long context into a set of latent tokens (e.g., 32 or 64 in length), achieving a high ratio of text tokens to latent tokens. A large LLM, as the second stage, then executes the decoding task on this compressed context. Experiments show that at a 20x compression ratio (where the number of text tokens is 20 times the number of latent tokens), our model achieves 98% decoding accuracy, compared to approximately 60% for DeepSeek-OCR. When we further increase the compression ratio to 40x, the accuracy is maintained at around 93%. This indicates that in the domain of context compression, C3 Compression demonstrates superior performance and feasibility over optical character compression. C3 uses a simpler, pure-text pipeline that ignores factors like layout, color, and information loss from a visual encoder. This also suggests a potential upper bound for compression ratios in future work on optical character compression, OCR, and related fields. Codes and model weights are publicly accessible at https://github.com/liufanfanlff/C3-Context-Cascade-Compression