mlsanityData to 14 Oct 2026

QwenLong-CPRS: Towards ∞-LLMs with Dynamic Context Optimization

arXiv 2505.18092 ↗First posted 23 May 2025cs.CLcited by 2

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance. Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference. Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59× context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 40 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

29 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar

Other references · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  12. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
16 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  4. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  6. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  7. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  8. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  9. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  10. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  11. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 119
  12. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  13. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  14. 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
  15. 2025Thus Spake Long-Context Large Language Modelcited by 12
  16. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  2. 2026ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RLcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  2. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  3. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Models
  4. 2026ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
  6. 2025An Empirical Study of Qwen3 Quantization
  7. 2026Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
  8. 2025QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management