# InfiniPot: Infinite Context Processing on Memory-Constrained LLMs

arXiv: https://arxiv.org/abs/2410.01518 · HTML: https://mlsanity.com/paper/2410.01518/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference](https://mlsanity.com/paper/2402.09398/) (2024)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference](https://mlsanity.com/paper/2403.20306/) (2024)
- [When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks](https://mlsanity.com/paper/2311.08993/) (2023)

## Built on by (most cited first, 27 of 27)

- [KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction](https://mlsanity.com/paper/2505.23416/) (2025)
- [InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding](https://mlsanity.com/paper/2506.15745/) (2025)
- [StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding](https://mlsanity.com/paper/2508.15717/) (2025)
- [Squeezed Attention: Accelerating Long Context Length LLM Inference](https://mlsanity.com/paper/2411.09688/) (2024)
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026)
- [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://mlsanity.com/paper/2512.03324/) (2025)
- [PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention](https://mlsanity.com/paper/2503.03588/) (2025)
- [Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing](https://mlsanity.com/paper/2502.12962/) (2025)
- [EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments](https://mlsanity.com/paper/2509.17396/) (2025)
- [NOSA: Native and Offloadable Sparse Attention](https://mlsanity.com/paper/2510.13602/) (2025)
- [Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits](https://mlsanity.com/paper/2511.00321/) (2025)
- [Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers](https://mlsanity.com/paper/2506.01215/) (2025)
- [Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction](https://mlsanity.com/paper/2510.20787/) (2025)
- [The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems](https://mlsanity.com/paper/2510.24831/) (2025)
- [QUOKA: Query-Oriented KV Selection For Efficient LLM Prefill](https://mlsanity.com/paper/2602.08722/) (2026)
- [KV Cache Optimization Strategies for Scalable and Efficient LLM Inference](https://mlsanity.com/paper/2603.20397/) (2026)
- [DySCO: Dynamic Attention-Scaling Decoding for Long-Context Language Models](https://mlsanity.com/paper/2602.22175/) (2026)
- [Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving](https://mlsanity.com/paper/2606.06302/) (2026)
- [END: Early Noise Dropping for Efficient and Effective Context Denoising](https://mlsanity.com/paper/2502.18915/) (2025)
- [Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge](https://mlsanity.com/paper/2602.22455/) (2026)
- [EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction](https://mlsanity.com/paper/2610.00412/) (2026)
- [APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention](https://mlsanity.com/paper/2601.21444/) (2026)
- [WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models](https://mlsanity.com/paper/2604.08558/) (2026)
- [Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines](https://mlsanity.com/paper/2604.16734/) (2026)
- [Context Recycling for Long-Horizon LLM Inference](https://mlsanity.com/paper/2606.26105/) (2026)
- [BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference](https://mlsanity.com/paper/2609.04971/) (2026)
- [Acoustic-to-Text KV Compression for Full-Duplex Speech Models](https://mlsanity.com/paper/2609.31224/) (2026)

## Abstract (arXiv)

Handling long input contexts remains a significant challenge for Large Language Models (LLMs), particularly in resource-constrained environments such as mobile devices. Our work aims to address this limitation by introducing InfiniPot, a novel KV cache control framework designed to enable pre-trained LLMs to manage extensive sequences within fixed memory constraints efficiently, without requiring additional training. InfiniPot leverages Continual Context Distillation (CCD), an iterative process that compresses and retains essential information through novel importance metrics, effectively maintaining critical data even without access to future context. Our comprehensive evaluations indicate that InfiniPot significantly outperforms models trained for long contexts in various NLP tasks, establishing its efficacy and versatility. This work represents a substantial advancement toward making LLMs applicable to a broader range of real-world scenarios.