mlsanityData to 7 Oct 2026

InfiniPot: Infinite Context Processing on Memory-Constrained LLMs

arXiv 2410.01518 ↗First posted 2 Oct 2024cs.CL cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Handling long input contexts remains a significant challenge for Large Language Models (LLMs), particularly in resource-constrained environments such as mobile devices. Our work aims to address this limitation by introducing InfiniPot, a novel KV cache control framework designed to enable pre-trained LLMs to manage extensive sequences within fixed memory constraints efficiently, without requiring additional training. InfiniPot leverages Continual Context Distillation (CCD), an iterative process that compresses and retains essential information through novel importance metrics, effectively maintaining critical data even without access to future context. Our comprehensive evaluations indicate that InfiniPot significantly outperforms models trained for long contexts in various NLP tasks, establishing its efficacy and versatility. This work represents a substantial advancement toward making LLMs applicable to a broader range of real-world scenarios.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

32 of 44 references.

References · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
20 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  6. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  11. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  12. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  13. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  14. 2024Long Context Compression with Activation Beaconcited by 101
  15. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  16. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  17. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  18. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  19. 2024TransformerFAM: Feedback attention is working memorycited by 21
  20. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 13.

27 papers.

Sort:

By area:

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  2. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  3. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  4. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  5. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  6. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  7. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 8
  8. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7
  9. 2025EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environmentscited by 7
  10. 2025NOSA: Native and Offloadable Sparse Attentioncited by 4
  11. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limitscited by 4
  12. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformerscited by 3
15 more
  1. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  2. 2025The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systemscited by 3
  3. 2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
  5. 2026DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Modelscited by 2
  6. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Servingcited by 2
  7. 2025END: Early Noise Dropping for Efficient and Effective Context Denoisingcited by 1
  8. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  9. 2026APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attentioncited by 0
  10. 2026Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edgecited by 0
  11. 2026WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Modelscited by 0
  12. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelinescited by 0
  13. 2026Context Recycling for Long-Horizon LLM Inferencecited by 0
  14. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inferencecited by 0
  15. 2026Acoustic-to-Text KV Compression for Full-Duplex Speech Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  2. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
  4. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  5. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts
  6. 2026Continuous Context Management
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
  8. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts