mlsanityData to 7 Oct 2026

Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity

arXiv 2502.13063 ↗First posted 18 Feb 2025cs.CL cs.LGcited by 27

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

A range of recent works addresses the problem of compression of sequence of tokens into a shorter sequence of real-valued vectors to be used as inputs instead of token embeddings or key-value cache. These approaches are focused on reduction of the amount of compute in existing language models rather than minimization of number of bits needed to store text. Despite relying on powerful models as encoders, the maximum attainable lossless compression ratio is typically not higher than x10. This fact is highly intriguing because, in theory, the maximum information capacity of large real-valued vectors is far beyond the presented rates even for 16-bit precision and a modest vector size. In this work, we explore the limits of compression by replacing the encoder with a per-sample optimization procedure. We show that vectors with compression ratios up to x1500 exist, which highlights two orders of magnitude gap between existing and practically attainable solutions. Furthermore, we empirically show that the compression limits are determined not by the length of the input but by the amount of uncertainty to be reduced, namely, the cross-entropy loss on this sequence without any conditioning. The obtained limits highlight the substantial gap between the theoretical capacity of input embeddings and their practical utilization, suggesting significant room for optimization in model design.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 57 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

36 of 57 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
24 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  4. 2014Distributed Representations of Sentences and Documentscited by 677
  5. 2015Generating Sentences from a Continuous Spacecited by 607
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  10. 2014Memory Networkscited by 354
  11. 2023Adapting Language Models to Compress Contextscited by 350
  12. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  13. 2021Efficient Attentions for Long Document Summarizationcited by 342
  14. 2015Skip-Thought Vectorscited by 321
  15. 2024Titans: Learning to Memorize at Test Timecited by 303
  16. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  17. 2023Representational Strengths and Limitations of Transformerscited by 132
  18. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  19. 2015Neural Variational Inference for Text Processingcited by 111
  20. 2020Memory Transformercited by 74
  21. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  22. 2021Sentence Bottleneck Autoencoders from Transformer Language Modelscited by 19
  23. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  24. 2024FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compressioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 13.

27 papers.

Sort:

By area:

  1. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  2. 2025Hybrid Token Compression for Vision-Language Modelscited by 19
  3. 2025Continuous Autoregressive Language Modelscited by 9
  4. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 9
  5. 2025REFRAG: Rethinking RAG based Decodingcited by 8
  6. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descentcited by 6
  7. 2025TokMem: One-Token Procedural Memory for Large Language Modelscited by 5
  8. 2025OSCAR: Online Soft Compression And Rerankingcited by 4
  9. 2025Exploring the Hidden Capacity of LLMs for One-Step Text Generationcited by 4
  10. 2026Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Modelscited by 4
  11. 2026Skill Neologisms: Towards Skill-based Continual Learningcited by 4
  12. 2025ARC-Encoder: learning compressed text representations for large language modelscited by 3
15 more
  1. 2025Sentence-Anchored Gist Compression for Long-Context LLMscited by 3
  2. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  3. 2025Learning a Single Token to Replace Long System Prompts in LLMscited by 2
  4. 2025Latent Reasoning via Sentence Embedding Predictioncited by 1
  5. 2025Compression is Routing: Reconstruction Error as an Intrinsic Signal for Modular Language Modelscited by 1
  6. 2026Concept Tokens: Learning Behavioral Embeddings Through Concept Definitionscited by 1
  7. 2026Progressive Cramming: Reliable Token Compression and What It Revealscited by 1
  8. 2025CoRoVA: Compressed Representations for Vector-Augmented Code Completioncited by 0
  9. 2026Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generationcited by 0
  10. 2026Hierarchical Embedding Fusion for Retrieval-Augmented Code Generationcited by 0
  11. 2026How to Utilize Complementary Vision-Text Information for 2D Structure Understandingcited by 0
  12. 2026How Many Different Outputs Can a Transformer Generate?cited by 0
  13. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  14. 2026AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generationcited by 0
  15. 2026Doc2LoRA Provides Decodable Representations of Scientific Ideascited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lossless Token Sequence Compression via Meta-Tokens
  2. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models
  3. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compression
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Models
  5. 2025Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
  6. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models
  7. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
  8. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation