mlsanityData to 7 Oct 2026

CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling

arXiv 2406.12018 ↗First posted 17 Jun 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Long sequence modeling has gained broad interest as large language models (LLMs) continue to advance. Recent research has identified that a large portion of hidden states within the key-value caches of Transformer models can be discarded (also termed evicted) without affecting the perplexity performance in generating long sequences. However, we show that these methods, despite preserving perplexity performance, often drop information that is important for solving downstream tasks, a problem which we call information neglect. To address this issue, we introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. In addition, we design a method for chunked sequence processing to further improve efficiency. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget, while preserving language modeling perplexity. The code and data have been released at https://github.com/ybai-nlp/CItruS.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

43 of 61 references.

References · 43

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
31 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  4. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  8. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  9. 2023In-Context Learning Creates Task Vectorscited by 302
  10. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  11. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  12. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  13. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  14. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  15. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  16. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  17. 2023System 2 Attention (is something you might need too)cited by 85
  18. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  19. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  20. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  21. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  22. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inferencecited by 32
  23. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  24. 2024TransformerFAM: Feedback attention is working memorycited by 21
  25. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  26. 2024LongHeads: Multi-Head Attention is Secretly a Long Context Processorcited by 16
  27. 2021RoR: Read-over-Read for Long Document Machine Reading Comprehensioncited by 15
  28. 2023Recurrent Attention Networks for Long-text Modelingcited by 12
  29. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  30. 2021Cooperative Self-training of Machine Reading Comprehensioncited by 5
  31. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Cognitive Memory in Large Language Modelscited by 28
  2. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  3. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancementcited by 10
  4. 2025Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning
  2. 2026ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution
  3. 2024Repeat After Me: Transformers are Better than State Space Models at Copying
  4. 2025REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression
  5. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  6. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  7. 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
  8. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving