# InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation

arXiv: https://arxiv.org/abs/2504.01707 · HTML: https://mlsanity.com/paper/2504.01707/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning](https://mlsanity.com/paper/2404.00213/) (2024)
- [Compressed Context Memory For Online Language Model Interaction](https://mlsanity.com/paper/2312.03414/) (2023)
- [Discrete Prompt Compression with Reinforcement Learning](https://mlsanity.com/paper/2308.08758/) (2023)
- [With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation](https://mlsanity.com/paper/2401.11504/) (2024)
- [Prompt Injection: Parameterization of Fixed Inputs](https://mlsanity.com/paper/2206.11349/) (2022)
- [Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass](https://mlsanity.com/paper/2411.05877/) (2024)
- [On the Way to LLM Personalization: Learning to Remember User Conversations](https://mlsanity.com/paper/2411.13405/) (2024)

## Built on by (most cited first, 17 of 17)

- [Reinforcement Learning via Self-Distillation](https://mlsanity.com/paper/2601.20802/) (2026)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Aligning Language Models from User Interactions](https://mlsanity.com/paper/2603.12273/) (2026)
- [Online Experiential Learning for Language Models](https://mlsanity.com/paper/2603.16856/) (2026)
- [Latent Context Compilation: Distilling Long Context into Compact Portable Memory](https://mlsanity.com/paper/2602.21221/) (2026)
- [Prompt-MII: Meta-Learning Instruction Induction for LLMs](https://mlsanity.com/paper/2510.16932/) (2025)
- [InfMem: Learning System-2 Memory Control for Long-Context Agent](https://mlsanity.com/paper/2602.02704/) (2026)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Context Distillation as Latent Memory Management](https://mlsanity.com/paper/2605.28889/) (2026)
- [LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks](https://mlsanity.com/paper/2607.18110/) (2026)
- [Doc-to-Atom: Learning to Compile and Compose Memory Atoms](https://mlsanity.com/paper/2606.12400/) (2026)
- [Understanding Parametric Knowledge Injection in Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12668/) (2025)
- [Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference](https://mlsanity.com/paper/2605.26099/) (2026)
- [Function-Space Temporal Self-Distillation for Policy Optimization](https://mlsanity.com/paper/2610.19687/) (2026)

## Abstract (arXiv)

In-context learning (ICL) is critical for large language models (LLMs), but its effectiveness is constrained by finite context windows, particularly in ultra-long contexts. To overcome this, we introduce InfiniteICL, a framework that parallels context and parameters in LLMs with short- and long-term memory in human cognitive systems, focusing on transforming temporary context knowledge into permanent parameter updates. This approach significantly reduces memory usage, maintains robust performance across varying input lengths, and theoretically enables infinite context integration through the principles of context knowledge elicitation, selection, and consolidation. Evaluations demonstrate that our method reduces context length by 90% while achieving 103% average performance of full-context prompting across fact recall, grounded reasoning, and skill acquisition tasks. When conducting sequential multi-turn transformations on complex, real-world contexts (with length up to 2M tokens), our approach surpasses full-context prompting while using only 0.4% of the original contexts. These findings highlight InfiniteICL's potential to enhance the scalability and efficiency of LLMs by breaking the limitations of conventional context window sizes.