mlsanityData to 7 Oct 2026

FlexComp: One Model for Every Ratio in Context Compression

arXiv 2609.11192 ↗First posted 10 Sep 2026cs.CLcited by 1

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Soft context compression condenses a context into a few memory tokens that a frozen LLM consumes in place of the raw text, but existing compressors fix the compression ratio at training and inference: each deployed ratio requires a separately trained model, and the chosen ratio is applied uniformly to all inputs, whose actual needs vary drastically. We propose FlexComp, a method-agnostic framework that decouples the ratio from both training and deployment: Matryoshka-style training samples the memory budget K per instance, turning one model into an any-ratio compressor, and the budget is then chosen per input by: (1) confidence-based cascade routing or (2) a lightweight learned K predictor. Across ICAE, 500xCompressor, and SAC on MRQA, a single FlexComp model matches separately trained fixed-ratio specialists with minimal degradation. Cascade routing preserves over 98% of the mildest ratio's accuracy at up to 266x average compression; the K predictor, in a single compression-decoding pass, reaches 158-236x within 0.7 F1 of the mildest ratio. At serving-scale batch sizes, the K predictor cuts context KV cache by 50% and improves decoding throughput by 47%.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 23 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

14 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors10 Oct
  2. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar

Other references · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  5. 2023Adapting Language Models to Compress Contextscited by 350
  6. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  7. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  8. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  9. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  10. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 59
  11. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  12. 2026EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commercecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
  2. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  3. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
  4. 2026End-to-End Context Compression at Scale
  5. 2026PoC: Performance-oriented Context Compression for Large Language Models via Performance Prediction
  6. 2024Characterizing Prompt Compression Methods for Long Context Inference
  7. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compression
  8. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM