mlsanityData to 14 Oct 2026

Vision-centric Token Compression in Large Language Model

arXiv 2502.00791 ↗First posted 2 Feb 2025cs.CL cs.CVcited by 21

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Real-world applications are stretching context windows to hundreds of thousand of tokens while Large Language Models (LLMs) swell from billions to trillions of parameters. This dual expansion send compute and memory costs skyrocketing, making token compression indispensable. We introduce Vision Centric Token Compression (Vist), a slow-fast compression framework that mirrors human reading: the fast path renders distant tokens into images, letting a frozen, lightweight vision encoder skim the low-salience context; the slow path feeds the proximal window into the LLM for fine-grained reasoning. A Probability-Informed Visual Enhancement (PVE) objective masks high-frequency tokens during training, steering the Resampler to concentrate on semantically rich regions-just as skilled reader gloss over function words. On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%. This method delivers remarkable results, outperforming the strongest text encoder-based compression method CEPE by 7.6% on average over benchmarks like TriviaQA, NQ, PopQA, NLUI, and CLIN, setting a new standard for token efficiency in LLMs. The project is at https://github.com/CSU-JPG/VIST.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 64 of its 87 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

64 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Mistral 7Bcited by 3,164
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  8. 2022Token Merging: Your ViT But Fastercited by 1,021
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  13. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  14. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  15. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  16. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  17. 2021OCR-free Document Understanding Transformercited by 407
  18. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  19. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  20. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  21. 2023Adapting Language Models to Compress Contextscited by 356
  22. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  23. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  24. 2022Memorizing Transformerscited by 226
  25. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  26. 2022Recurrent Memory Transformercited by 203
  27. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  28. 2023Augmenting Language Models with Long-Term Memorycited by 189
  29. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  30. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  31. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  32. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  33. 2023Retrieval meets Long Context Large Language Modelscited by 115
  34. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  35. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  36. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  37. 2022Parallel Context Windows for Large Language Modelscited by 81
  38. 2022Language Modelling with Pixelscited by 55

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  2. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  3. 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
  4. 2026ReGuLaR: Variational Latent Reasoning Guided by Rendered Chain-of-Thoughtcited by 10
  5. 2025See the Text: From Tokenization to Visual Readingcited by 9
  6. 2026VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoningcited by 8
  7. 2026Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compressioncited by 5
  8. 2025OutDreamer: Video Outpainting with a Diffusion Transformercited by 4
  9. 2025OmniGaze: Reward-inspired Generalizable Gaze Estimation In The Wildcited by 4
  10. 2025EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimizationcited by 3
  11. 2025ARC-Encoder: learning compressed text representations for large language modelscited by 3
  12. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reductioncited by 3
9 more
  1. 2026Cross-Modal Memory Compression for Efficient Multi-Agent Debatecited by 2
  2. 2026LensVLM: Selective Context Expansion for Compressed Visual Representation of Textcited by 2
  3. 2026Decoupling semantics from vision: A framework for faithful visual-text compression evaluationcited by 2
  4. 2025Optical Context Compression Is Just (Bad) Autoencodingcited by 1
  5. 2026Spectral Evolution-Guided Token Pruning in Multimodal Large Language Modelscited by 1
  6. 2025H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classificationcited by 0
  7. 2026LoMo: Local Modality Substitution for Deeper Vision-Language Fusioncited by 0
  8. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domaincited by 0
  9. 2026FocusVTC: Efficient and High-Performance Visual Text Compression with Adaptive Resolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
  2. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods
  3. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
  4. 2025FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression
  5. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMs
  6. 2026Global Context Compression with Interleaved Vision-Text Transformation
  7. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
  8. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models