# Glyph: Scaling Context Windows via Visual-Text Compression

arXiv: https://arxiv.org/abs/2510.17800 · HTML: https://mlsanity.com/paper/2510.17800/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [RWKV-7 "Goose" with Expressive Dynamic State Evolution](https://mlsanity.com/paper/2503.14456/) (2025)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Extending Context Window of Large Language Models from a Distributional Perspective](https://mlsanity.com/paper/2410.01490/) (2024)

## Built on by (most cited first, 30 of 49)

- [AgentOCR: Reimagining Agent History via Optical Self-Compression](https://mlsanity.com/paper/2601.04786/) (2026)
- [Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning](https://mlsanity.com/paper/2601.14750/) (2026)
- [Context Cascade Compression: Exploring the Upper Limits of Text Compression](https://mlsanity.com/paper/2511.15244/) (2025)
- [MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning](https://mlsanity.com/paper/2601.21468/) (2026)
- [VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?](https://mlsanity.com/paper/2512.15649/) (2025)
- [ReGuLaR: Variational Latent Reasoning Guided by Rendered Chain-of-Thought](https://mlsanity.com/paper/2601.23184/) (2026)
- [EfficientPosterGen: Semantic-aware Efficient Poster Generation via Token Compression and Accurate Violation Detection](https://mlsanity.com/paper/2603.00155/) (2026)
- [VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning](https://mlsanity.com/paper/2601.22069/) (2026)
- [VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?](https://mlsanity.com/paper/2602.04802/) (2026)
- [Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs](https://mlsanity.com/paper/2603.09095/) (2026)
- [Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression](https://mlsanity.com/paper/2602.00746/) (2026)
- [FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space](https://mlsanity.com/paper/2602.02092/) (2026)
- [OneLatent: Single-Token Compression for Visual Latent Reasoning](https://mlsanity.com/paper/2602.13738/) (2026)
- [InnoGym: Benchmarking the Innovation Potential of AI Agents](https://mlsanity.com/paper/2512.01822/) (2025)
- [From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition](https://mlsanity.com/paper/2512.14244/) (2025)
- [When Text-as-Vision Meets Semantic IDs in Generative Recommendation: An Empirical Study](https://mlsanity.com/paper/2601.14697/) (2026)
- [Proxy Compression for Language Modeling](https://mlsanity.com/paper/2602.04289/) (2026)
- [VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understanding](https://mlsanity.com/paper/2602.10146/) (2026)
- [MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs](https://mlsanity.com/paper/2602.18600/) (2026)
- [WebNavigator: Global Web Navigation via Interaction Graph Retrieval](https://mlsanity.com/paper/2603.20366/) (2026)
- [LensVLM: Selective Context Expansion for Compressed Visual Representation of Text](https://mlsanity.com/paper/2605.07019/) (2026)
- [PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.28344/) (2026)
- [DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents](https://mlsanity.com/paper/2608.01827/) (2026)
- [Decoupling semantics from vision: A framework for faithful visual-text compression evaluation](https://mlsanity.com/paper/2608.01848/) (2026)
- [Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression](https://mlsanity.com/paper/2608.02109/) (2026)
- [Optical Context Compression Is Just (Bad) Autoencoding](https://mlsanity.com/paper/2512.03643/) (2025)
- [Global Context Compression with Interleaved Vision-Text Transformation](https://mlsanity.com/paper/2601.10378/) (2026)
- [ImgCoT: Compressing Long Chain of Thought into Compact Visual Tokens for Efficient Reasoning of Large Language Model](https://mlsanity.com/paper/2601.22730/) (2026)
- [Document-as-Image Representations Fall Short for Scientific Retrieval](https://mlsanity.com/paper/2604.18508/) (2026)
- [Hard to Read, Easy to Jailbreak: How Visual Degradation Bypasses MLLM Safety Alignment](https://mlsanity.com/paper/2605.07250/) (2026)

## Abstract (arXiv)

Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-language models (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph.